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FOREWORD 



The research described in fhis final vep&n was conducted by 'the Human Resources 
Research Organization. Sponsorship has been from Grant GJ774 from the National 
Science Foundation. Additional support for the activities was provided by the* Depart- 
ment of the Army (Work l^nit IMPACT and CATALIST). 

The work was performed at HumRRO Division No. 1, Ale.xandria, Virginia. Dr. J. 
Daniel Lyons is Daector of the Division. — 

In addition to Dr. Seidel, principal contributors to the research activities have been 
Mr. Richard Rosenblatt,^ Dr. Edward Schneider, .Mrs. Judy Corftmon, Dr. John Stelzer, 
Mr. Michael HiUelsohn, Vnd .Mr. William Underbill. The report was prepared by Dr. Seidel, 
Mr. Rosenblatt, Mr. HlLlel^ohn, and Dr. Stelzer. 
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SUMMARY 



The purpose of this National Science Foundation grant w*as to conduct pro- 
grammatic research in the area of instructional decision models '^DM). This /was 
accomplished over a three-year period in a* computer-administered instructional (t?AI) 
environment. The prii^ipal sponsor of HumRRO's CAI research and development efforts 
during that time was Ihe Department of the Army (Army Research Office), under an 
overall program entitled Project IMPACT. The research conducted under this NSF grant 
augmented the IiMPACT efforts. 

Th^ research strategy followed cyclical or iterative development and testing of 
successive v«:sions of instructional decision models. This permitted the improvement of 
. decision rules by which the subject matter, COBOL, was taUght in an individualized CAI 
environment. Two iterations of the COBOL course w6re, designed and developed. The 
latest IDM made possible a more indivi^lualized, adaptive, much improved course of 
instruction. It should be noted that the findings herein reported were gathered using an 
operational course vehicle with real-world application. The average time to complete the 
instruction was approximately 60 hc5urs. 

Also developed as a result af this research strategy were a number of software 
support capabilities that could be generalized to other individually /Adaptive instructional 
environments. A total sys^nis approach was u^ed in which the strategies, software, and 
content, as well as hardware capabilities, were revised from cycle to cycle during the 
course of the research. 

Pre-course histwies of individual students were incorporated as potential predictor 
variables^ as were the students* in-course" histories. Factors included in this exploration 
were Structure of Intellect tests developecl .by Guilford and his associates (said tests being 
relevant tD cognitive kinds of tasks such as^ the COBOL course represented) and the use 
of Level of Aspiration or expectations of the students concerning their^otential, as well 
as some exploration of the use of some confidence measures as additional in-depth 
assessments of the students' state of understanding, " • ' 

Parallel to the efforts at Jmproying the understanding of the students' attribute 
structure, with regard to entry ch^ractenstics and within-course changes in states of 
understanding, was the continued empirical* and formal development of subject-matter 
structures. Techniques for modularizing course content were developed and implemented 
on-lihe within the HumRRO environment. General subject-matter mapping rules were also 
formalized during the conduct of this research, and this work continues beyond the 
tenure of 'the grant* , ; 

The.IDM research results veriffed, in general, the^value of Structure of Intellect test 
batteries in differential prediction of student performance with diffcfing task requi^ 
m^nts in the class of course content exemplified by COBOL (e.g., problem solving tas^ 
of a high cognitive and memorial variety), ' - • * 

The value of student control options in a total tutorial CAl' environment was also 
explored as part of this research effort. Essentially, the data- showed that there are 
differences in the use of' control options by^stuclents who tend to be "high performers in 
contrast to those who tend to be low performers. The characterization of high performers 
was shown to be unique to the particular tasks or levels of the .course and not a general 



trait characteristic for a student across all portidns of the course. Indeed, these 
characteristics were shown to be relevant and describable by the entry characteristics tests 
used during the research. ^Generally speaking, the high performers tended to use the 
student control options to a lesser degree and with greater efficiency than did the low 
performers. 

A raimber of important cOiarse developme^nt tools were also accomplished as a result 
of this research effort. They, provide major by-products .to the National Science Founda- 
tion and\an be generalized to other computer-instructional environments. Some of the 
tecJhQKiu^ developed have already been ad€>pted elsewhere (e.g., da^a management and 
text-logic separation techniques taken on by Uthe TICCIT project at Brigham Young 
University). These techniques are described within thfs document and in the supple- 
mentary appendices. They include such things as preformatting capal^ilities for authors to 
use- with minimal requirements for computer programming expertise, automated trouble 
reports that can be used on-line by proctors, arid improved Coursewriter recording and 
analysis functions, as well as the general approach to modularizatiori of course content. 

Dissemination 'of information on the progress of this rese^ch effort has been 
accomplished through prcbcntations at various professional and scientific meetings, as w^ll 
as in publication form. These piroducts are listed ^iii th4j.final section of the report. 
Additional dissemination \\?ls been accomplished in respond to requests for the opera- 
tional C0B0L2 course, now available to any interested users. This use of Jhe research 
recults is discussed in> Section 2, under Accomplishments, , v 

The ' remainder of this report is separated into five sections relevant to Xhe 
deliverable products promifod as a result of th^ research activity. S&ction 2 is an 
introduclion which deals v.ith the objectives and accomplishments; next is a section on 
'the details of tlfc IDM rch?arch, followed by a section on the COBOL ^course imple- 
montntidn, along with the colirsj development procedures and tools. Sectiop/5 provides 
overview of^ the software developments desicned to support IDM research and 
operational • implementation of couri^es of instruction, along with various formative^ 
evaluation techniques drawing upon the computer for efficient iterative development.' 
Included here are data 'analysis techniques, Coursewriter function descriptions, and the 
various reporting capabilities developed at HumRRO. Lastly, the listing of and brief 
description of the publications accomplished during the conduct of this research effort* is 
provided. The appendices include brief summaries of the more relevant supporting 
documents, which arc available in separate form. 
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Section 2 
INTRODUCTION 



OBJECTIVES 



The research proposal prepared by th^ Human Resources Research Organization 
addressed the problem of developing a comprehensive instructional model through an 
iterative process. Our approach distinguished betv^een two avenues toward achieving^this 
goal. Empirical tests were iised^to identify the variables of importance for prediction in 
the learning environment. In parallel, theoretical design was used to provide the formal 
framework within which the instructional m^^iel can^e formulated. 

The empirical work therefore ,^vas concerned wi£h testing* versions of the IDMs. 
Through extensive testing, measurement, and evaluation in a, learning environment, two 
successive versions of the IDM were implemented, evaluated, and modified. This process 
led to the refinement of the IDM. At the same time, a theoretical framework in which 
the results of the EDM research could be coherently described was developed^ 

The general goal of the effort under thfe National Science Foundation grant was to 
provide a model of the human learning process. The model must be descriptive in that it 
must provide a precise description if5f the learning process. The model niust also be 
prescriptive in that it must provide for specific solutions with respect to (Optimizing the 
learning process in the management of instruction. Conceptually, when the prescriptive 
portions of the learning model are implemented &s rules or procedures for controlling an 
insthictional process, then an Instructional Decision Model (IDM) results. The descriptive 
portion of the model yields constraints on the IDM. In this sense, the descriptive and 
prescriptive portio.ns of the model merge in both the operating IDM., and the course 
vehicle. * ^ ^ 

Since the initial course vebicFe (COBOLl) and- the initial IDM seemed uncjuly 
restrictive,^ a rapid and massive effort at revision \yas made. The result^(COBOL2) and 
the Interface schema^* provided at the end of two years some developmental products 
originally promised for delivery to the Foundation at the end of the third year. Empirical 
research results, in turn, were provided at the end of the third year instead of the second 
year as originally planned. / • " ,^ 

The 1970-71 Tec^inical Progress Report reviewed the approach taken and reported 
on the fact that less empirical research, work had been done than originally planned 
during the first year and a halt. The reseairih was prinqipally devoted tAv^tablishing 
systems reliability and implemenfe'g the course vehicle. While this action rekilted in 16ss 
data collection and experimental manipulation of pertinerit variables, the available tim^ 
permitted a substantial! amount of (a) conceptual development relevant to a model for 
individudKzed instructib'n, and (b) empirical testing and refineirfent of course development 
products. Both of these efforts, described in separate documents (siee Appendices A and 
B), ^ have provided powerful tools for more * efficient research on the descriptive/ 
prescriptive problem of relating the learning model to the instructional dcci.sion process. 



See Technical Progress Report, 1970-71. 
? See Proposal for Continuing Support, January 1972. 
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This work was performed during the third year of the research grant in order to answer 
the questions of how student and "system control could best be implemented to facilitate 
management of instruction in an adaptive, tutorial CAI environmenL— 
In summary, the proposed, objective of this research was to: ' 

(1) Arrive at an appropriate model of the instructional decision process for a 
given class of content ^ 

(2) Implement the IDMs as computet4)rograms. * 

(3) Ultimately determine optimization procedures for a meaningful CAI 
; ^ program. \ ^ ^ ^ . . 



ACCOMPLISHMENTS 




Instructional Decision Model (IDM) • ^ - 

The implemented version of the first IDM has been described in the 1970-71 
Technical Progress Report (pp.6, 7, and .1449). Initial results of the Iteration 1 testing 
of, the model were also described in that document (pp. 17-18), 

In general, firm conclusions concerning the adequacy of the initial IDM are not 
possible, for several reasons. First, the confidence** measure as a diagnostic tool could not 
be evaluated because . of faulty implementation. As has* been indicated previously 
(Technical Progress Report, 1970-71), students tended, over time, to operate with the 
'principle of least effort, using stereotyped cbnfiderice estimates (in the main, 0 Qr 100%). 
• Secondly, because of the limited ni^ber of students used (N total of 42) and because.the 
^ course proved to be very difficult (mean percent correct in final -criterion, 59%), the data 
analyses could reveal only £entativQ suggestions for further wdrk. Further analyses have 
been conducted smce the last report was submitted. These analyses used better clustering 
techniques which eliminated bias in obtaining centroids in the previous analyses and 
provided additipnal step-wise multiple regression results. 

Given the above limitations, the suggestions from the data obtained in the first two 
years were in essential agreement with the previous work of Guilford, Bunderson, and 
associates (Guilford, 1967; Bunderson, 1967; Dunham andl Bunderson, 1968; Dunliam, 
Guilford, and Hoepfner, 1968) concQmirig the applicability of Structure of Intellect 
concepts and of differential performance predictors useful in tasks of different levels of 
complexity. The research during the third year verified these findings apd suggested that 
specific student profiles were associated with ability to self-ipanage instructicy|.^Moreover, 
the. studertt characteristics 4:orrelated with high performance were uniqu^to sp<5cific 
divisions of the revised COBOL course, again supporting ihp concept of differential 
performance transfer based upon a match bet\^een instructional tasks and student 
descriptions. These findings and recommendations for future research are discussed in 
detail in Section 3 of *this repoi:t, IDM Research. ^ ' ' 

The requirement foj extensive course' revision and formative evaluation" came at^ a 
point in time which was simultaneous with a reduction in overall funding from the Army, 
a principal sponsor of the FIum-RRO CAI research. The result was a constrained 
implementation of new decision rules as they p^jrtaincd to self-assessment (expectancy 
operators and the use of improved confidence measures).' Howeverf a redesign of these, 
rul^s was completed and they^ are considered to be useful guicU'Hnes for instructional 
strategies V.'herever self-ass^jssmSnt and use of niotivational indices m-e desirable. These 
decision rules are discussed for the use^ of Leve'l of . Aspiration (LOA) and confidence, 
measures in Section 3 of this report. . ^ ' 

The "limitation .on equipment and personnel resources permitted only the , first 
crxperiment, .cm student versus ^system control, to ho complotcd. The comparison of 
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figural versus semantic presentation of materials is currently being a<^gned for 
implementation during the latter' part of FY 1974'' and the beginning of FY 1975.^* 

The formal theoretical model under development is at present, primanly ^descriptive 
in nature and concerns ^ssentially the mapping of subject-matferJ^OTucbu're with the 
example provided so far, principally Booleap, algebra. In additi<5n to this, Appendix A 
presents a preliminary formulation -of ^model to reprgsenfstudent attribute structures. 
Prescriptive „ guidelines are currently being jdeveloped for^ the managemer\,t of an 
individualized instructional environment, vis-a-vis unique instructional strategies matching 
student attribute structure with the structure of the subject matter. During the course of 
the formal* theoretical work, the reduction in funding from the Army sponsor necessi- 
tated, seeking additional support to accomplish this portion of the res^earch. To this end 
funds were sou(^t and obtained from the Air Force, and some empirical tests of 
subject-matter structuring wete applied to>^a short Air 'Force course. The work governing 
I the progress in formalizing the instructional process is detailed ^an Section 3, 
lf)M Research. ' 

In brief, the Iteratioti 1 of the Instructional Decision Model suggested that self- 
assessment could be a useful tool for prescribing management ofLjthe instructional process; 
however, more work needsrto.be done in this area. The Level of Aspiration (expectations 
on the part of the student) be quite useful as a predictor for student performance. 
Howev^, the use of cpnfidcnco measures was not implemented adequately enough to 
permit final jucjfjment on its utility io augment the use of expectations with respect to 
self-assessment.' The confidence implementation was redesigned .fn some off-line imlial ' 
testing with staff ^jnembers, and fpjidolines have been d^yelop^d for^its inclusion in a 
subsequent jn^ructional strategy investigation, h can be used, it is felt, lis an aid to 
, assessing progress in inst^ri^ction and as a prescriptive ipdex tof remediation or acceleTa- 
tiorj^ However, no fi^rm jud[jments could be*made from our existing research. 

The second iterfition of out IDM incjudpd the modularizatioh of the course materials 
in accordance with specified subsets of behavioral objectives, (see Appendix C as "an added 
detailed discussion of this^conccpt). Abo, the Interface IDM framework was introduced* 
and used quite effectively, along with the testing of various student or system controls 
over the options for remediatfon and - acceleration. As indicated, these findings are 
discussed in more detail in Section 3, along with recommended use of our iDM-findings. 

Course Development 

* During the course of development' of instructional decision models, we also 
developed an operational COBO^i course which is described in Appendix; B attached to 
this final rejfcrt. The total number of students used to Validate the COBOL courserHhat 
is, in terms »of development, revision, jind retesting leading to an operational product— was 
205. COBQLl involved 42studonts7 and C0B0L2 included 83 students testing the 
revision,' of the instructional decision model, plus an additional 80 s^idents who consti- 
tuted the experimental sample for the research on student versus system control options. 

Compared to COBOLl, C0B0L2 reorganized the materials, that is, introduced some 
gaming and some restructuring, ^particularly the modul^ization already^ described. 
Secondly, the performance 'was much irpproved over CO BO 1^1, especially with the last 80 
students who served as the experinij^tal subjects. Again, tlriif will be discussed in more 




"1 ' • • 

^This^wiU bo accomplished on a now terminal system, Plato IV, sponsored !)y the U.S. Army 
Research Institute for Behavioral and Social Sciences (ARI). HumRRO lost its .systini capability thiring 
the latter pWt -<3f ^FY 73* An additional huidiance has been thi' delay in /lciiv(ry of required new, 
hardware; thi^s oxpferiniclitation has had to 'bo postponed. 
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detail in Sections 3 .and 4 (die latter conjcernihg course development and in^plementation 
procedures). Suffice it tp say here that in the revised CQB0L2 course, for"" the intro- 
ductory part of the course (Director), the first quartile score was 90% of Qbjeptives 
attained on first attempt, the median scorQ was 80^^, and the'lhird quartile \vas 75%. For 
E(ivisi6n B, 'the second part of the course, the experimental students again scored an 

>vel^ge of 80% conect in terms, of "number of " objem;iVes passed" the firsi time (the 
median); first quartile in Division B was"85% of objectiv^ passed the first time, and the 
third quartile was 75%': These data are in stark confra§tsto the initial criterion level of 
59% fo*CC\BQI4. Thus, the course was much improved af^gr revision. ^ ' 

In addition, the ac^coitiplishments under course develo^ent consisted of upgraded 
development and evaluEftiqn procedures with an automated ^ trouble report capability 
(VOYEUR Program for proctoi:^^, modularization techniques in the.fonp of SUPER- 
EDITOR Preformatting ^capability, as Wellj^the techniqtit^ for dividingJJ^e materials into 
subsets of behavioral course objeq^tives^^^J^xt, the data rhanagenipnt''capabil!ties described in 

' this report in Sectiop^S (particulairly iPES-2) v/^re ^upgraded to be much more efficient and 
relevant to the classic storage ^and retrieval capabilities of standarc^ data processing 
systems. Finally, there were improved recording and analysis functions 'developed to 
interface .with our CAI language, 'a modified version of Cdursewriter III of the IBM 
processing system. ^' ^ ' - , ' ; ' 

A copy of the operational course is designated as* Appepdfx D/ l^e coAtptete coursfe 

^pacKage includes a mierofiche of text, student manual, course logic, and the intxoductoiry 
,coufse. ^ : . ^ ' > , . , ' . 

Disserrfination ' . ' / ' ' . ^ y ^ 

. ' ' ^ ^ ^ ^ . ' ' . - 

*With the completion of the operational C0BOL2 course, a number of requests have 

been received, and are in the process of b^ing fulfilled for documentation aiad "course 
listing in order that the course can be given elsewhere. To date,, the Rochester -Technicals 
Institute for the Deaf has requested and been sold, at C9st,^a complete copy of the 
COBOL, course. Othef 'recjuests'Sn process have conie from the U.^! Census Bureau, Ohio 
State University, and SiniofN Eraser University. 'A complete package of all course listings 
and documentation h|s also been forwarded to U.S. ^Continental Army Comrn&nd (now 
called TRADOC). Ballou H^h School of the District of Columbia hasi impWented a 
computer-managed (CMI) version of the course. , ^ - 

Additional dissemination of our materials has. been accpmplished thro,ugh the request 
and delivery of software techniques ani^ome' listings to the JJrigham Young University, 
Instructional Technology Group. (A letter of appreciation from t)r. C. Victor Bunderson 
is attached as Appendix JE.) ' / It ' ' 

All of the course products are available from'Divison Nb. 1, HumRRO, at cost. 
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_ " The second version the Instructional Decision Model was implemented in tha 
COBOL course cbncurrently- with the/large -scale revision, described elsewhere, of 
that material.^ ' ' ; * ' n . ^ ^ , 

Principal among the features of 'the nemlDM ts the ^ability to afford the student a// 
specified degree of control over the sequencing of inslnictional material. Of the four 
"tleamer-eontrol" optiQos used in fhe study, th^ee-REVIEW, RECAP, and Ql%-affect 
r^nfedial activity^ amh-acceleration; the fourth,^ ROUTE, " consols topic presentation ! 
sequence at specified points in the course-. " \> 

Specifically, should a^tuclent type QUIZ hp is branched^from hiB currfeht location in 
a topic to the beginning of the ^Quiz*' sectioil^G^TthgrtopiG (attempted lise of this 
control option, or any of the other three for that master, while in the Quiz section' 
would be ignored by \bie IDM). If the^ student types UEVIE^ he is r^umed to the 
beginning of the topicr if he^ types RECA?, he is shown a list of tiie^topics he has ahready 
taken and allowed to review as many as he wishes without^^^aving^ r^ke the quifzes. 
The stu(Jent may exercise the first three options" at his discretiori/'ThKfourtl^ option, 
iCOUTE, is ena^bled by the IDM at any point where^lhe course };jrerequ^te s^c$ure 
perfnits selection from equivalent topics. At such points the student is shown a ^^memt^ 
of topics presently available, o'rfe of which he may choose to enter' next; if he declines to 
choose, tjie system will rsfcdomly picl^' one of the^topics for himl ' * 

'These four learner control" onfionc nnnnnr\<:o^ iua it^^o^o 
present experiment.^ 

During th^ conduct of the experiment, tliree types of measures were taken on^ach 
entry charapteristips, ^including aptitude,,.affective, and bfographical data; learner 
j^s, including type arid fr^^quency of control ^sage and the circumstances of their 
use; ami^chievement>and other *T3erfoi;pa^ce^eflate4 measures including quiz scores, 
transit time^Drogramming errors, opinions of topics, an^ Level of Aspiration prior to^the 
. quiz of each tbpic.^ 

It was the fn^rpose, then, of the present experiment to ^aSess-th^^lative contribu- 
tions and interactions among tfie learner-control and entry characteristic "v^^ables with 
respect to instructional effectiveness and efficiency as represented 'Sy the 
dependent measures, 

EXPERIMENJAL -DESIGN 

Prior to, the start of the *cours^^ach^tiia(ifiKwa^ randomly assigned to one of the 
16 possible /combinations of -the leaSb^^control options; thus some students had all 
options available, sonie hadHKicess to nol^, some had Snly one of the four, while others 
had some combination of/two or three control 6ptions. . i ^ 

Every student saw /the, same topics, aftl^ough not always in the sanle orde^ since 




options comprise^ the independent variables in "ttie* 



those wl^ had ROUTE weife able to modify 



e sequmce to some extent. EUrther,.the 




\\ • ■ • 

particuj?ur option combination 'to which students were assigned and the extent to which 
they exercised those options would cause the exposure to .a' gi\&en topic to vary 
among students. \ • 

Students who failed the quiz for a giv^n topic were branc^hed back to the beginning 
of thajt tppiC" to restudy the njaterial ^nd retake those quiz items previously missed; the 
amount of time spent in restudy depended on the student^s own discretion and on his 
option combination (e^., a student with ttie QUIZ option could jump to it immediately,. 
\VH^reas,a student without QUIZ Would pei^forcAsee all instruction in the .topic before his 
second attempt at the quiz items), ^hould the ^tudent fail the quiz a second time, the 
same procedure would be followed except that a\ staff member would monitor Tiis ^ird 
attempt at the quiz and clear up any misconceptions evidenced, by the answers. 

At the end of each division of the coyrse, the students wei;e required to code, run, 
and debug a COBOL program to demonstrate^ast^ry of the skills taught in the topics of 
that division; successful completion of this task Was a prerequisite to starting the topics 
of the next division. ' ' \ 

The revised COBOL course originally contained four such divisions with a total of 
32 topics. Comprising about 60 hours of instruction. reduction in resources coupled 
with the impending Joss of HumRRO's in-hpuse cort^puting capability necessitated 
shortening the cour&fe in order to guarantee a sufficient number of subjects for the 
experiment; Consequently, for the present study the fiW two divisions,' comprising 
21 mpdules^nd about 30 hours of instruction, were used. ' i 

SUBJECTS ^ 



Ninety percent* of the sample (N=80) were paid ^^Qlunteers recruited through 
advertisements in the local newspapers; the remaining subjects wai::e^militaj!'y personnel, 
also volunteers, suf^lied by the Army*s Project Transition— afl activity desigied to assist 
^ separating soldiers in developing job-relate)cJ skills for civilian life. . 

Our experience in the first COBOL course demonstrated that students who w^ 
severely deficient in programming aptitude were generally incapable of acquiring even the 
basic skills taught in the course. Since such students provided little usable data while 
placing an additional strain on our already limited resources, we screened out any 
prospective student whose score on the IBM Programmer Aptitude Test fell below a raw 
score of 46 (a "low C") by more than one standard deviation. 

To make our experimental .findings relevant to real world training, we intended that 
our subjects reflect the characteristics of programming trainees'^generally— young, with a 
minimum dfsome high school, and naive with respect to programming. The data indicate 
that these requirements were met. Table 1 summarizes the 'relevant biographical 
char^^cteris tics of the sample.^ 

^'^ghty'-two percent of the sample were 30 years old or younger; 52% were 25 or 
younger and kpprp^cimately 217c were under 21. Ninety-six percent had no programming 
9 experience or training. ^ ' 

Nearly 99% of the sample had at least some high school education. Seventy-six 
percent had at least a high school Education, and 55^^ had at least some college, while 
30% had a college degree or beyond. 

While we consider the age spread of the sample to he appropriate, they were, if 
anything, somewhat overeducated for our purposes; the effect of their schooling on 
course performance will b^ shown in the results section. , ^ ' 

le over half the sample had vision difficulties (generally a need for glassc^), in 
response to^an exit questionnaire almost no orte reported any probjems reading the 
cQUrs6 materials from the- display devices. 
#■ , ' . ■ 

- ■ . . -10 - • ^ 
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Parsohal Data for Experimental Subje. 



Perc^nt'of 
Responses in . 



Percent of 
Total Qroup 




Typing Speed - 
-Non-typist 
1^20«VPM 
^ 2J'40WPM 

m -80WPM 
/ Greater than 80 WPM * 
' No response 

Educational Level 

Eigbth/grade years)' 
Some high school (9-1 1 years) 
Completed high school (12 years) 
Some college (13- 15 years) 
Completed college (16 years) 
Graduate (more than 16 years) ' 
;;>>.^er 

Training and Experience 
f^^ . ^ No training or experience 

''*^^ome^ training and/or e)^perience 

f' ^^^Jlf^"^ Occupational Stajtus 
• ? Stuctent 

Military , . 
Employed-other 
Unemployed 
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DESCRIPTIOINJ OF ENTRY CHARACTERISTICS MEASUftES 

Student entry chaxactensties were measured in the following four etreas: Stii'ucture of 
Intellect Factors, /Motivation, Computer Programming Aptitude^^^^^and ' Reading 
Comprehension. ' \ : ' ^ ' 

The Entry Charactferistics Test (ECT) battery ^consists of 27 in^urnents .wl)ich. yield 
\ 35 distinct scores; all but^one are time tests. They range .in lengtn^of testing ^ime from 
2 minutes to 70 minutes, with a majority of the t6sV involving *less. than 20 minutes 
testing time. In general^ tlie tests' are of the paper-atnd-pencil jCariety and designed for 
administration in group testing sessions. 

Structure of Intellect Factor Tests ^ 

These tests measure ^ten factors and were tests/ used most recently by 
I967)^here are 27 tests used for factor measi^ement. The factors and tests 
used to measuretReSF^actors are described below. • 

(1) General Reasoning (2 tests). This factor has been described as *'the ability ^to* 
solve a broad range of reasoning problems, including those of a mathematical nature" 
(French, Ekstrom,'and Price, 1963). These tests selected to define this factor are the Ship 
Destination Test (Sheridan Psychological Service^,* Inc.) and Necessary Arithmetic 
Operations (Educational Testing Service), . ' . - 

(2) Induction (3 tests). This factor' has been described as "associated abilities: 
involved in the finding of general concepts that will fit sets qf data, the forming and 
trying,^out of hypotheses'' ^French et aL, 1963). The Letter Sets Test (Educational 
Testing Sorvice),* Locations^ Test (Educational Testing Service), and Figurp Classification 
Test (Educational Testing Service) define this factor. ' ,\ 

(3) Fi giiyal Adaptive Flexibility (5 tests). French etaL describe this factor as "the 
ability to change set in order to meet new requirements imposed by figural problems/' 
The following five tests define this factor; • ... 

(a) Match Problems IV (Part 1 and Part 2) (Aptitudes Research Project,* 
University of Southern California). 

' (b) Match Problems V (Educational Testing Service). 

(c) Word Coding Test (designed by Lennart Sjoberg, John Frederiksen, and' 
Victdlr Bunderson). ^ ^ 

(d) Decoding Test (designed by Lennart Sjoberg, John Frederikseji, and " 
Victor Bunderson). 

. - -*(4) Verbal Reasoninc ^ (3 tests). This factor has been given a number of different 
names, including ''Deduction" by Thurstone, "Logical Reasoning" by Guilfor<l, and 
^'Syllogistic Reasoning'* by French a/. (1963). French a/, describe it as "ability to 
reason 'from stated premises to their necessary conclusionc/' Tkn^ Nonsense ffyllogisnls 
Test ' (Educational Testing Service),' Logical Reasoning Test^(Sheridan Psychological 
Services* Inc), and Infer?hce Test (Educational Testing Service) wore selected to define 
this factor. 

(5) Symb6l Substitution (1 t^st), Guilford and Hoepfner (1966) classify this factor 
as a factor of convergent production and* define it as "the ability to produce a 
completely determined, symbolic deduction from given symbolic information, where such 
an% implication has not been practiced^ as such.'* One test defines this factor: Sign 
Changes (Aptitudes Research Project, University of Solithem" California). 

(6) Chitnking Memory (1 test). This is a new factor postulated by Bunderson/He 
designed two tests; in the present study one of them, the * Binary Digit Span Test, 
npeasures this factor. He also developed a control test, th(^«Numbcr Span TestA^S, which 
is "similar in content hxii not in opportunity for chunking" (Bunderson, 1967). 



(7) Memory Span (2 tests); This factor has been described as *Hhe kbiliX td 'recall 
^ perfectly for immediate reproduction a series pf ftems after only one pres'^'tatfbn of the 

series" (French a/., 1963). The marker tests for this factor are the AuaitofyWumber 
Span Test (Educational Testing Service) and the Auditory Letter Span Test (EdiA^ational 
Testing Servip^).' ' > ^ ^ ^ 

(8) Associative Memory (3 tests). This factor is defined ^ the ^abil'it^ to reirfember, 
paired associates. The. three tests which defirxe this factor are: the Picture-NumbeA Test 
(Educational Testing Service),' the Object-Number Test (Educational Testing Service)\and 
the first and Last Names Test (E(iuca|ipnal Testing Service). \ 

(9) Perceptual Speed (Iv test). 'This factor is ^described as the ability to m\k^ 
comparisons and find figures fast and accurately. The 'test which defines thisVfactor is the' 

■" Num^ber Comparison Test (Educatipnal Testing Service). ' V 

(10) Spatial Scanning (1 test). This factor is defined as "speed in visually exploring 
^ Wide OE^omplicated spatial field"' (French a/., 1963). The Maze tracing Speed Testf 
:(Educational Testing Ser/ice) measures it. . ^ / 

Motivation * * 

Included in the Entry 'Characteristics Test battery are four tests which were selected 
to measure anxiety and achievement motivation. The psychological literature is replete 
with studies showing relationships between anxiety and learning in laboratory situation! 
(Spence and Taylor Speiice, 1967). Recently studies by Hansen aijd associates {1969} 
.have also shown some v.-ylue in the study of anxiety as it relates to performance in CAlf 
'These three tests are the IP AT Anxiety Scale Questionnaire ^Institute for Personality and 
Ability Testing), the.SaraSon Task Anxiety 'Questionnaire (adapted frpm Mandler and 
Sarason, 1952), and the Sentence 'Completion Test of ' Achievement Valuel/ 
(Mukherjee, 1964). ' ^ * ^ ' )/ 

Computer Programming Aptitude ^ ^ ^ , // 

A survey of the literature^ revealed that by far the most widely used test of aptitude 
for programming has been^' IBM's Programmer Aptitude Test (PAT) and -Reviled 
Programmer Aptitude Test (RP AT),, A large body ^ of reliabilityoand validity datA is 
associated ^vith these tests. 'Recently, th^ PAT and RPAT> have' been replaced byfme 
Aptitudfe Test f6]^ Programmer Personnel (ATPP), which is included in the battery^This 
.test correlates highly with both PAT and RPAT. , 

A second' test. Primary Mental Abilities (PMA)/;\viir also be used to njLsure 
programmer aptitude. The Primary Menfel 'Abilities Test Ijas for .years been used /by the 
RAND Corpc^ratron ^and System^ Developjnpnt Corporation: as"" a, programmer selection 
^device (Perry and Cantleyt 1965; ^owen, 1^57). The Army uses a programming aptitude 
test developed' by a civilian company-andT'yery similar to the ATPP. While the t^t is not 
included in oxir &attery, fhe^ ^'scores for military subjects were obtdiied for 
research purposesr^ ^.-r'^ ^ ' ' ^ . 

Reading Compression ^ ^ \* ^ 

— Fin^lyj^^e Entry Characteris^^ Test Battery includes one test of reading cornpre- 
hen^o^/6he Reading Comprehension .Cooperative English Test (Cooperative Test Division 
o^^r-fitmcational Testing Service). ThjjLinstrument provides four scores: vocabulary,* level of 
-^^^eading comprehension, speed of reading comprehension, and total reading comprehension 
' (level + speed/2). ' ' ' > ^ . 
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RESULTS 



The depepden^ measures ggthered in the experiment were initially analyzed by 
means of the .Analysis of variance; Figure 1 contains the underlying statistical model with 
annotation ot each variable. Because it fs a mixed, moc^el with nested factors, expected 
mean squares \ver^ osculated; these and the F ratio^^propriate to testing each effect 
are shown in Figure 2. 

The analyses of variance Were computed on quiz scores for the first try only and on 
tiransit times for the first an^ second tries; beyond th^se limits, most of the cells of the 
design would be either empty or of unequal size. 

The results of the analysis are presented 'in Table 2. Notice that the only statistically 
significant main effects (p<.05) occur for the units of instruction. The very large F ratios* 
obtained for all transit times, and the smaller but significant' ones for quiz performance, 
compilation errors, and LOA simply indicate that the instructional \jnits varied in th^ir 
difficulty and in their length, 'flhe few significant interactions that involve the learner 
control options are due to the fact, reported in the experimental design .section, that the 
extent to which students make use of the options affects the amount.©^ time spent in 
a topic. 



Dependent Measure 



Transit Tiaie 
Telling/Practice 
Section {1st try) 

Transit Time 
Quiz Section 
{1st try) 



Total Transit Time 
All Sections 
{1st & 2nd tries) 

Transit Tjme 
Divisions 




Table 2 

nalysis of Variance Results 



^ifficant Main Effects 
& Interactions 



Ratio P 



Quiz Score 
Compile Errors 
LOA 



Top^"* 

Route X Quiz X Topics 
Topics 

Route X Topics * 
Review X Tbpics 
Qui? X Topics 
Route X Quiz X Topics 

Topics 

Recap X Review 
Review X Quiz , ^ 

^Review 
Review Xiaij[iz 

Topics 

Divisiona 

Objective ^ 
Route X Recap - 
Route X Objective ^ 
Recap X Quiz , ( 
Route >^Revrew'X*^Qul2 
Review X Quiz X Objective 



Degrees of 
Freedom' 




5.88 
1.86 

254.40 
2.07 

1.68 
2.07 



20;1280 
20;1280 

20;1280 
20;1280 
20;1280 
20;1280 
2O;i280 




y 



Statistical Model llnderly/ng a 27 x n Repeated 

Measures Factorial Design . ^ 

^ijklmno i j k 1 mdjkl-V » ^i^ ij ik- iJL_ 

^ + af. + be, c*- bd.!, + bf; i cd, - A- cf, + sf * 
in . jk ^^3l • jn \^ kn m(ijkl)n ■ 

In 

+ abc. + abd. + abf . . + acd / + ajcf 

ijk i3[l ijn xkl ^^ikn 

+. adf + bed., T + bcf + cdf, - 
^ . - iln .jkl jkn kin 

' 

4- abed..,- +abcf.., 4-acdf., ^ +bcdf.,- +abcdf^.,, 
- ^^'i., _ ijkl_ ijkn ^ ikln jkln ijkln 

\ % • ""^^ 

oCijklmn) * . * ^ . 



a^ = route option, where i = 1,2 ^ 
^ b^ = recap option, where" j =1,2 
cr^ = review option, where k = 1,2 

= quiz option, where 1 = 1,2 
^m(ijkl) ~ students, 'where m= 1,2,3,. ..,5 



^n ^ instruction segments, where n = 1,2 i(course divisions) 

V 

n'= 1,2,3»...21 (division modules) 

I ^ 

. ^, n ' * = 1,2,3, 26' (module o6ject4ves) 



o''(ijklmn) ^'^^^P^^^^^^^^^l ^rror, where 0 = 1,2, 3^ •••,80 

/ ■ ■ v.. •, _ 

/ Figure f 
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Expected Mean Squares ipr thp Statistical Model 

Underlying a 2^ x rj, Rejieated Measurer Factorial 
Design (where, n = T,2,3, . . . ,26) ' 



Mean Squares for Computing Observed^"?"' Ratios 



a 


0^ + n80o^^+ n320<k7^ 


, MS >f« 
a s 




e . s - . b 


no, / no 

b s , ; 


c 


2 2 9 '"'^ 
0 + n80a + n3200a i. 
e s ^ \ 


V HS /MS 
> ^ c s 


d 

• 


ol + n80a^ +'n3?00a?^ 
^jAi^ 


- MS /MS x; J 

d s ^ , y - ' 


s 

e ^ — 


2 2 
a +'n80a > 
c s . 


^ Testablp 




f 


+ 80a + 64000^ 

sf f 


HS-7MS 

. f sf ^ 

>^ - 




+ n80a^ + nl600a\' 
s Tab 


MS , /MS 
ab s 




2^2 2 
a; + n80cj + hl600a 


MS /MS 
ac s 




ol + n^0a^ + nl6i)0a^, 
c s ad 




■ 


'ol + 80af. 3 2000^ • 


MQ ' 7mC '^V 

af^^^s/- \ 




— + n80a^ +,nl600a^ 
c ^ s* . be 


/ ; 1 : — : 

jUo, / no V, 

DC S ^'v.^.^u./"''^ 




• ' 2 . ' 2 'r 
0 + n80a^ + nl600a^, 
c s •bd 
— - 


MS /MS 

bd s 


:^ bf 

i 1 


ol + 80af + 3200(T^^ 
c V sf bf 


MS /MS 
* bf^^^sf 


^ c'd 


+ n80af + nl600a^^ 

^ S QQ "* 


t MS ,/MS 

Ca s ' *■ 


cf 




MS ./MS ^ 
cf sf 


• sf 


2 ? 
+ ^00^. 

G sf 


Not Testable 


df ' 


+ 80a^^ + 32000^,^ 
e sf ' df 
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Expected Mean Squares for the Statistical Model 

Underlying a 2^ x n Repeated Measures. Factorial 
Design (wherein = 1 ,2,3 26) (Conjtinued) ' / 



c • .abc 



MS , /MS 
abc s 



e s abd 



MS , ,/MS 
. abd s 



abf 



MS . _/MS , 
abf sf 



2 2? 
e ' s acdi 



MS ^/MS 
acd s 



e sf aqf 



MS ./MS ^• 
acf ' sf 



e s bed 



+ 8Da^^ + 16000^ ^ 
,e sr bcr 



MS,^./MS . 
bcf sf 



^oj + 80af^ 4- 16000^ 



cdf 



MS ,./MS . 
cdf sf 



e s abed 



MS , ,/MS 
. abed s 



i + 80af^ + 80«o\^^ 



MS, - JHS ^ 
abcf sf 



a + 800 8000? 
£• sr ^cdf 



^^Ncsp^a ---1: * 



^^6cdf/^^sf 



ol ^'St^l, ^ A00a^\^/^ 



abcdf sf 
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. Absence" of significant effects of any option on quiz performance implies that ^the 
Independent variable made no difference in the learning. Examination of the' overall 
performance by the students suggests why these results occurred. Figure 3 indicates that 
the great majority^ of the students did quite well in the cgurse. Notice that the frequency* 
distributions of .objectives passed on first try for both Divisions A and B are severely 
truncated with a marked positive skew. Seventy-five percent of the students -passed at 



Frequency Distribution of Subjects, by Percentage of 
- Learning Objectives Passed'on First Try 




least 70% of the objectives on the first try for Division A; fully half^the students passed 
75% of the objectives on the first try. The results for Divisioi^B are comparable. 'First 
quartile scores for Divisions A and B axe 9P% and 85% respectively. Third quartile'scor^ 

* are identical, 70%. ' r L 

. These data strongly suggest that the .limited^ariability it student performance, 
CQupleit with a restricted'range and severely skpjvi^ distrilbuti'on rendered a conventional 

'.technique such as the analysis of variance hi^propriate £k a means of analyzing the 
effects of the independent variables. Such ajr Dccurr^ce is not uncommon in educational 
research, where the instruction^ materiaj/used as vehicle^ for experimentation must also^ 
irledt the reqtti^:ement that ^£ teach well. . / * 



HIGH AMD LOW PERFdtjMERS 

*It was -decided, that, under the cjircumstajices, a potentially fruitful way ot investi- 
gating the jeffects o| learner control ^would be*through= a comparison of the best and ih^A 
worst students in the sample for any treatment .differences (i.e.. in tiie use of the-^ 
options). The **high/l&w" perfornier technfque used in the.dcvelooment of psychological 
inventories (Brennan, 1970) was applied. The individual items i^Q analyzed for their 
capacity to discriminate between those whose overall test :cf res are high and those whose 
scores are low. In the present ^study we wish to analyze the way learner conltrol options 
are used, as disc^miriators"^ high and low perfom^ers in the COBOL course.^ \ - ^ 

The following^questions'were posed Yor this aspect of the analysis: ' \^ 
^ s {!)' Who are the high performers in Division A of the coujse? Who* are the ^ 

loly performers? 

(2) Who are the high, performers in pivision B? Who^^uieJiow^xEormer^?^ 
^'•'(3) Dp they differ in the way or -manner .iij^^Vmch" they make_us6 of the 
options?' 



(4)^Is there any difference. t! 



ibutes of. the high and low performers 




which might helpT explain or pVedict^ifferences in option use? 
Selection 

In brder to have a, sufficient nymber of observations for the^a^^^c^isri^^ decided 
that the 20 highftt and lowest |)erformers in each division woujS^beJdemuied ior further 
study.* A combination of absolute and relative perrormancc-ertferia were used to select 
them; the specific criteria for high and low performers in earh division are shown in 
TahJe 3. " ^ - . ' . ' / 




Manner of Option JUse 

The frequency *with which the Tiigh and^loW^^p^foirme^rs used the options is 
presented in Table 4. The values have b^^ adjusted* to equate option* availability across \ 
groups. This was necessary because,Jn^some instances, the high and 'low performers did 
not come from treatment ^conditions having the same degree of access to options. 
(Because second attem'pts occurred with relatively low frequency, these data on option 
lisage ate not reported here.) Data- on the ROUTE option usage are presented in Table 5. 

Table 4 shows that the low performers consistently use the options more frequently . 
' than the high performers in both divisions. This suggests that, if the options aid learnings 
at all, the gain is due not to how often i,hey are used but rather to where iind when they 
are used. ^ ' ' *•> *^ , V 



Hrabre 3 

Selection Criteria for High and Lciw,PeFior 

(Divisions A and B ofjba-eOBOVCourse) 



lers 



A 



High Performers on Divisigp^A 

Criterion 1: Passed >JB4^69^^>>f first-attempt objectives. 
Criterion 1Zi Of first-attempt passed objectives, 72.7% 
Criterion 3: Top 20 ranked according to score 
"obtained S, 



we 



100 percentile. 




d) 



, max obtained S,^ 
Low Perforrpers on Division A ' 

Criterion 1 : Failed > 30,8% of first-attempt objectives. 

-Criterion 2: Of failed' first-attempt qjjjectives, 75.0% were < 23.75 percentile] 
Criterion's: Bottom 20 ranked according to scpre 

* / ^ / obtained S, 

I score = ^ ' 



High Perfornr^s on DivisioCi B' 



max obtained S, 



)) 



Criterion 1: Passed > 84.6% of first-attempt objectives. 
Criterion 2: Of first-attempt passed objectives; 66.6% were > 98.75 percentile. 
Criterion 3: Top 20 tanked according to score ^ c ^ * 

obtained S, 



/ _ / obtained S, \\ 

I score = 2 ( — ] ) 

\ \max obtained S,// 



Low Performers on D ivisicm . B 

Criterion 1 : Failed > 23.1 % of first-attempt objecjhfes. 

^Criteriorv 2: Of f if st-atterr^t failed objee^a^>75.0% had scores < 26.25 percentile. 
.^Criterion 3: Bottom 20 ranked according/to store 

Y obtained ^ 
\max o^t^ned Sn 

I 



l^core = 





Table 4 

-requeney of First Attempt Option Usage ^ 

(Adjusted) 



20 



24 



Performers 




Optio^ 








RECAP 


REVIEW 

SJ^ 


' QUIZ 


Division A 










• 


High 


6.2 


12.0 


' 15.0 ' 




0 


" Low 


35.0 ^ 


65.0 " 


25.0 






Division 6 




/\ / 








High 


15.0- 


. 4.0 


8.8 






Low " 


20.0 - 


52.5 


. 11. > 
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Tabte 5 ^ 

Use of Ihe. ROUTE Option by 
High and Low Performers 



Performers 


• 

ACTIVE 


PASSIVE 


, TOTAL 


Division A' 










High 


f 


*t4 


5 


19 


Low 




10 


•10 ^ ' 




Division B 










. iligh 


^ 21 




32 


Low 




18 


19 ' 


37 . c 



* Students who haJ the ROUTE option, coul3, when presented with a "menu" of 
available topics, choose to pi.c^t their own or defer, to* the system to pick one ^t random. 
Table 5 showe that' the ROU^E option Was made available by the JDM far more, often ih 
Division, B than in Division A; tt^is occurs because the prerequisite' structure in Division B* 
is less ordered.^ In both 'diyigions, the higH and low performers saw menus about the 
same numbef'of times. However, the proportion of occurrences in vi^hich subjects made 
an active selection of the next' topic differs markedly between the hi^bs and lows. In 
Division A the' high perform'ers made thfeir own clioice*nearly three times more o/teh than 
not; thj?^ low performers actively chose only half the time. In Division B the high 
performers (?hose nearly twice as often as they deferred to the system; here, the lo<v 
performers also -chose about half tlie time. ' 

Personal Attributes / r ' ' 

In an* effort to identify ihe cognitive and affective characteristics on which high and 
low performersf differ, the .'Entry Characteristics Test scores of j;hese groups jwere 
subjected to multiple stepwise discriminant analysis. The results are sumn>arized in 
Table 6. 

The overairdism^inatim-^ both divisions was highly, significant (for A the overall 
>was 10.154; for E( tjjeyi^ was 9.609;' p<.01),-'In Divisioi^A three of 'the 35^entry 
characteristics score^wer^ selected by the analysis: th^ Primary^ Mental Abilities test,' a 
measure of general verbal aptitude; Match Probleihs, Test V, a measure of figural adaptive 
flexibility;, and ihe IP AT Alixiety Test^core B, a measure of the extent to wjiieh an 
individual reports anxie.ty-related feelings or behaviors. Examination of the standardized 
coefficient^ tor these variables shows that » the greater part of the discrimination is due to 
the verbal abilities test. The positive sign of the • coefficients indicates that the high^ 
perforniers possess these attributes^o a greater degree than the low performfer^. ^ 
^ For Division B ^he analysis identified prpgrammin^ aptitude, the Aptitude Test for 
I^ogramming Personpd, and vobabulary, measured \yy the Cooperative Enghsh Test, as 
the principal ^discriminators between the high andMoW performers. Two tests— the PMA 
Spatial Relations and the Letter Sets Test (a measure of inductive reasoning ability)— are 
not easily, interpreted in the present context. The negative sign of .the coefficients mean^ 
^the high performers possess less of these attributes thaij the low performers, although 
one would expect a positive relatidti with success in the couitge. 

, - " / 1 

' See Figures ^ and 9. . * ' ' * ^ . 




. ^ ' ^ ' Tables 

Summary of Stepwise MuWplk^Jj^criminant 
Analysis of High and Lo)Vyerform6rs on 
35 Entry Characterisffcs Test Scores 



Division 




^Test Selected 



Standariiirzeci 
Discriminant Weight 



bal / / 
ietyt^core "B" 



PMA-Verbal 
IPAT Anxiet\ 
Match Problems V 

• Signfifitac(c^ of overall discrimination: 
f (3,36) = 10.154; p«, 01 

Cooperative English^Voc^ulary 
•AtPP ' 

PM^-Spatial Relations, ' > 
} Letter Sets . # 
' First dnd Last Names , 

£;/gnificaru:ejjt,o\t^alJ_djsa:lmina^ 

f (6,33) =9.609; p«. 01 ' 



7.738. 
1.560 
2.506 



19.742 
.13^14 
-6.57 
-2.693 
3.007 



1 
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SELF-ASSESSMENT 
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Two techniques .ol self-assessment as potential decision factors ^ere studied in this 
research effort: expectmcy (LOA) and copfidence measures^' - ' ' 
/ * ' 

Expectancy Measure I' ' ^ . * * '* 

During the fii^^it iteration of U)M investigatioh, LOA (Level of Asplratioi^ was , 
sttidipd as a cormlatiohal vai'iable. LOA conelated sfgnificantly with criterion per- 

.formance in both. /evels of complexitx. % COBOLl. The correlations werq +.42»and +.53 
(p<.01) respectively. These findings^ were consistent with previdus results in programmed 
instruction (see ^Mdel a^d Hunter; 1970). j ' 

It was th'erj&fore decided to make LOA part ofi the decisiori-paking strate^ in the^ 
revised IDM ick COBOL 2/ Extensive course revision coupled With curtailed*" resources 
prevented infimementation of LOA as- the newly developed Expectancy Operator, bulo 
tiOA was usra again- as a correlational variable. The results were cohsistent with the 
previous' d^,ta,' indicating significance student expectations as predictors of achievement. 
Becaus^of the modularization %i C0B0L2, it was ^possible to perform a« finer-grain 

'analysis thaji previously. TheJ^OA data WQre analyzed at* the level of specific objectives, 
^he J)asic Jiypothesis tested was tha^ high performeife/vvoUld be more realistic than low 
performers: Operationally this would take the form of (»)>a smaller positive discrepancy 

, score for the higlT performejrs, an^d (b)r^ smaller absolute value qf ^fscrepancy^ between 
LOA an4',objedtive score for the high^perfocmers, ^ . 

A '^eiling effect because of the eiccellent •performahcev.by the Irigh . performers 
prevente^ an-j an^ysis of the signed differences. However,^he test of absolute value 
differencesS^Gyealed in Figure 4 clearly supporte& the hypothesis that bigft* ^rformers 
would be more, realistic than the low performers. Coup)ed with the previous LOA 
finding^ these results substantiate the value of providiitg an Expectancy Operator for 

^ remediational purjjo.SjBS as part of an. improved IBM* * 



The fact that the LOA measures indicated a hi^' degree o/ value to self-assessment 
andihe fact that other studies /e.g., Shuford, Albert, and^Mass^ngill, 1966) supported the 
value of confidence measures as an aid to learning led us to re-evaluate the ways in \vhich 
\v£ would implement confi4ence"* measures'' in C0B0L2 (rather than eliminating 
confidence as a sensitive index of state of understanding). ' . ^ 

The g9al in C0P0L2 was to make the implementation easier for the student to use, 
make all" input responses equivalent in effort and difficifity, lessen the frequency with 
^^hich the confidence measures were used to avoid interrupting and interfering with' the 
yleaming process, and lastly to increase the value of providing confidence measures by 
making associative materials attached to the various states of understanding more 
meaningful and positive than they had been for the student in COBOLl. 

The redesign was accomplished and initial off-line preliminary testing wa^ achieved 
with staff members of the research project. However, because of the limited resources 
'land other difficulties cited earlier, the re-implementation of confidence measures was not 
accomplished during this reseai'ch project. We feel, nevertheless, 'that, m combination with 
the Expectancy Operator as 'discussed above, the confidence measures should provide a 
very sensitive component to revised deqision-making rules taking into account student 
motivation. The suggested implementation of confidence measures i^ provided as follows. 

Confidence testing would be part of the Q-sections of the course, COB0L2, and 
handled in the following manner. ^ * i 

For the constructed response type of question, the student,' after making yare that 
his answer is the one he wishes to have recorded and checked, will input his response. His 
display (CRT, hard copy, etc.) will be cleared and^ a confidence, question will be 
displayed. This confidence question will summarize the task ask^ed of the student and ask 
\m to place his confidence in a prescribed location on the display. * / ; 
* Tlie student's confidence willvbe indicated by his selection of one of Jl characters 
from his keyboard: The pharacter^ are 0, 1, 2, 3H,'-5, 6,^ 7, 8, 9, aVih where T stands 
-for 10. A 'computer program ^computes 'the number of joints :tBe- stud.ent receives by 
multiplying the number ''of points a question is \Vorth (10^-99), as determined by the 
i author, by a ttiree-pl^ase ^decimal associated with the student's confidence (see Table 7). 
Fojr a correct 'response, the Student receives thaj/ number o|' points. If he is incorrect, he 

' ^ - ' ^ Table 7 

> Scoring System for Confidence Measure 
• • * ^» 

f 



Student Inputs for 
Correct Alternative 


Payoff 


0 


0.000 ' 


1 


0.500 


2 


0.650 


3 ' - 


0.739 


4 


0.801 


5 


0.850 


6 


0.889 


7 


0.923 




0.952 




0.977 


T 


1.000 
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The rationale behind^ tl^ Expectancy Operator, is based upon^the relationship 
between a student s perforniafujfean the relative reality of his Expectan>si^. If a student 
is judged to be unrealistic-^toor^^at a discrepancy between LOA and p&rfgrmafi^ 
Probe path analysis and remediatibn ^re B^v^ted. Recommended instru^jojial guidelines 
for initial use of the ExpectancyT>peratoraii^ probe path are as follows. ^ 

It is suggested that in^plementation of% tfie" Expectancy. Operator (Ld*A) take the 
form of providing for LOA measurement prior to a Q-section, criterion test on specific 
subsets of objectives, and measurements of student performance for each module (subset 
of objectives). The IDM takes specific action based on a comparison of these 
measurements. The action to be tajjen by the IDM is based upon a decision of a problem 
in rea//f>" estimation or conceptual understanding, or some combination. In essence, the 
(discrepancy score relationship b^Jbween the LOA difference score in combination with 
percent correct (*and when refined, confidence) is to be used for determining whether or 
not a Probe path should-t)e followed. A Probe path is initiated by the system in order to 
gain more information concerning the student's problem and in order to take appropriate 
remedial action. 

For example,^ if, following the pretest, the alignment of realism and percent correct' 
represents "appropriate" behavior,, then the student ,is to follow the "fiormal" path for 
him. (Normality in this case 4* to be defined idiographically.with a continually refined 
entry battery.) If discrepancies occur in the student's estimations of reality', then the 
Probe path will be- followed foi; additional diagnosis and action (by ^noting the relation- 
ship bet\Veen the measures of percent coinrect and the measures of the IX) A). 

The sjJecific plan and guideline' for initial implejnentation of an Expectancy 
Operator in a tutorial environnrjent follows. ^ ' 

1. Objective for using the Expectancy Operator: To lessen the relative distance 
between level of aspiratiqn^d performance by raising the level of the lower 

« (LOA or performance) to jneet the^ higher .value. 

2. Plan for measuring LOA and pmviding solutions: ' ' . 

a. Measure LOA prior to entering the Quiz. ^ 
h. Measure performance in the Puiz. 

c. Eeeiiback to student on LOA versus performance, verbal plus nujncri- 
cal comparison. . ' " ^ t. 

d. Provide solution. ' 

* (1) Solution to raise performances to^ LOA: Follow-up alternate 
strategy with alternate quiz (check discrepancies). * * 

(2) Solution on subsequent modules to bring confidence level up to 
{DerfprmAnce: Check discrepancy 'scores on subsequent modules. 

3. State DIAGfilOSES: Eighteen possible states are derivable from the three 
'sets of characteristics listed below. For purposes of initial implementation it 

is suggested that>3a, A3b, Cla, Clb will be most useful as Additions to an 
^ Captive IDiM. * ' ^ 

LOA r , PERFORMANCE ' ' ^ ABILITIES 

A High ^ 1 High ^ a Above Average 

B Average^ ^ 2 Average .. b Below Averagd. 

C Low 3 Low 

4. Set of Alternative ACTIONS: 

- . a. Remedial modules geared to specific eContent failures qoncentrating on 
variation of practice exercises. 

b. Conference with proctor/instructor. 
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c. Fun Option (on-line games). 

d. Skip aliQad (practice). 
L eavin^^ 

f. Alternate media module (CMI type, cassette, PI text). 



g.- (Jompliment. 
"^TTT^Dcrnothing. / * . . 

5. Recommended ACTIONS for DIAGNOSES (A3a, ASlS, Cla, Clb): 

c, d, e, g: Confidence Building (Cla, Clb) 
a, b, f: Performance B.uilding (A3a, A3b) . 

6. Actions to be implemented initially: 

• * » A3a: a ^ . . " . . 

^ • A3c: b and/or f 

' Cla: d, e, f, and/or g 
Clc:^orh 

' General illustrations of how theTPribe path would operate *are as followsfVirst^ let 
E stand fpr the student's estimate of his^formance after the fact; letL stand fo/his 
level of aspiration or expectancy befo^^ the fact; and lefA stand for his ^actual 
^performance. The definition -of j is^ tridl number, of a referent fdr the particular' 
' moasurenrient nuipber of LOA or EST. Applied to c6B0L2, it would reference module 
\\number. Generally,, the Values for L,E, andA would be derived^ from j:he degree of 
^crite^ion attainment determined for a particular application in a given computer- 
based environment. ^ ' ^ 

A: DIAGNOSiSr If ILj- Ail> lEj- Ail&(Ep^=+) V 
then student state is defined as REALISTIC to the IlDM. x * u , / ' 

ACTION: Diasnosis proceeds to next stage in IDM. S^ore on Test of 
Objectives, percent correct, defines gONCEPTUAl state and confidWe->^ue defines 
REASSURANCE state. Givea^A, and passage 'of criteria here, student continues on 
"normal" module path availablV (based on cuAent options— eventually to be redefined by 
our improved Entry Battery ^nd' better within-course historical predictors). SiJbject 
possesses the three R's— Realistic, Reassured, and Right /(see Figure &) 
"B. DIAGNOSISr If tLj- Aj l< lEj- Aj I (E^isk^_j ^ ' 

ACTION: Go to PROBE path. ' ' ' 

PROBE paffit Here IDM can be thought of in the following way. 
(1) It can query student directly to determin.e the nature of tbe piroblem as 
;perceived by the student; that is, the student ^ays, "No problem," or "I don't think I can 
hack it;" or "I think I understand this stuff, but I'm not sure" (or'some variation on^ 
this theme). . . r • ^ 

* (2) Diagnosis* of problem is defined as the int^ectiorp'of three orthogonal 
binary dimensions. .The resulting state is estimated ^s follow/ (verbally t)elow and 
"^pictorially in Figure 5): /^^t. 
\ ^ ' (a) Given B above (diagnosis of UNREALISTIC) an^i^^WConfidence and 
High Objectives scQre, then two dimensional motivational \pfebkms exist— 
REASSURANCE and REALISM. * 

(b) Given B, arid Low Confidence and Low Objective^ -Score, tii^ 
problem is diagnosed as both overall motivational and CONCEPTUAL. ; 

dc) Given B, High Confidence and High Objectives Score, then problem-is 
uniquely one of REALISM (e.g., the pessimist ^ven tbough confident at time answering)., 
' (d) Given B, High Confidence and' Low Objectives score, then also 
REALIS^ but probably of different type (e.g., delusions of grandeur). 

o (e), (f), and(g)^ Cases currently handled by the IDM where the 

E;xpectancy Operator woujd't^e in the zero condition. 
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Illustration df Diagnostic States 



Subject is REALrSTiC, REASSURED, and RIGHT 




High Low 
Reassurance ^ 



Figure 5 



Mathematically (and for IDM use) the state diagnosis can be described i>y ordered 
triples where l = a problem condition, remedial ^operator /is called for, and|0 = ri6 
i problem. Thus, reading Realism, Reassurana^, and. Conceptual dimensions fr<5)m left' 
to right: ^% y ^ 

Case (a) is described as <1,1,0> ' ^ ^ 

' Case(b) <1,1,1> * , • 

Case(c) <1,0,1> r 

'>CasV^(d) . ' <1,0,0> 

Cases (e), (f), and (g) <0,1,0> : <0,1,1> .<0,0,1> 

Confidence Measure ' " 

A second measure of assessment used in our IDM research effort was confidence 
responding by the §tudents. In COBOLl, unlike the implementation of LOA measures, 
confidence responding was part of every student response. The s^tud6nt gave an answer to 
a question and immediately thereafter distributed his confidence with respect to the 

^ answer over a series of alternatives if the alternatives were available; or he attributed a 
degree of confidence £o the correctness of the answer he provided in a completion 
type^format. ^ » 

The results, as reported in the Technical Progress Report, 1970-71, indicated a' 
lowering of the c<^rrelation^^tween confidence measures and con-ect responding as the 
student progressed through %e 18 modules of COBOLl. The implementation scheme 

^^' ^ apparently was not a useful one for the students. They were required to give a percentage 
..^v^e^etween 'fX^md 100% using two digits as appropriate (e.g., 45, 55) and eventually 
aadptefCa^principlT?of iQast effort. That is, the students either used a 0 or 100% 
confidence ..choice eventually, ^d the result was a. lessening of the value of the 
confidence measure as an indicator^of a state of understanding on the part of the student. 
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The fact that the LOA measures indicated a high' degree of value to self-assessment 
andihe fact that other studies Xe.g., Shuford, Albert, and.Massjgngill, 1966) supported the 
value of confidence measures as an aid to learning led us to re-evaluate the v^ays in vy:hich 
we would implement confi4ence'' measures'' in C0B0L2 (rather than eliminating 
confidence as a sensitive index of state of understanding). , . ^ 

The g9al in C0P0L2 was to make the implementation easier for the student to use, 
make air input responses equivalent in effort and difficiiky, lessen the frequency with 
i^hich the confidence measures were used to avoid interrupting and interfering wiih' the 
yleaming process, and lastly to increase the value of providing confidence measures by 
making associative materials attached to the various states of understanding more 
meaningful and positive than they had been for the student in COBOLl. 

The redesign was accomplished and initial off-line preliminary testing wa§ achieved 
with staff members of the research project. However, because of the licnited resources 
*^and other difficulties cited earlier, the re-implementation of confidence measures was not 
accomplished during this research project. We feel, neverthelesSr, that, m combination with 
the Expectancy Operator as 'discussed above, the confidence measures should provide a 
very sensitive component to revised degision-makihg rules taking into account student 
motivation. The suggested implementation of confidence measures i^ provided as follows". 

Confidence testing would be part of the' Q-sections of the course, COBOL2, and 
handled in the following manner. ^ * , i 

For the constructed response type of question, the student,' after making ^re that 
his answer is the one he wishes to have recorded and checked, will input his response. His 
display (CRT, hard copy, etc.) will be cleared and" a confidence, question will be 
displayed. This confidence question will summarize the task asked of the student and ask 
\un to place his confidence in a prescribed location on the display, * 
* T\\c student's confidence wilkbe indicafed by his selection of one of il characters 
from his keyboard: The pharacter^ are 0, 1, 2, 3; 4;^5, 6,. 7, 8, 9, a^h T-, where T stands 
* for 10. A Computer program ^computes 'the number of f)oint$*:tRe studjent receives by 
multiplying the number ""of points a question is \Vorth {lff-99), as determined by the 
I author, by a tbree-plajpe , decimal associated with the student's confidence (see Table 7). 
Foj: a correct 'response, the Student receives thaj? number o| points. If he is incorrect, he 



Table 7 

. Scoring System for Confidence Measure 



Student Inputs for 
Correct Alternative 


Payoff 


0 


0.000 ' 


1 


0.500 


2 


0.650 


3 ^ - 


0.739 


4 


0.801 


5 


0.850 


6 


0.889 


7 


0.923 




0.952 


9 , 


0.977 


T 


, 1.000 



27 



receives the number of points :found by multiplying the three-place decimal associated 
with the ten's' complement of the student's confidence by the author's point value for 
the question. ^ - 

For example, if the, student places a T in his confidence block, then the ten's 
complement is taken as his **no confidence" response, in this case, 3. Suppose the author 
states that the question is worth 60 points; then, if \the^ student is correct he receives 
60x0,923 = 55,3 rounded to 55 points, ' and if he is incorrect he receives 
60 X 0.73&= 44,3 rounded to 44 points. 

Each of the computer point, values rolls'onto the display in the proper location. K 
the*sClhient is dis^tisfied with the^ number of points he will receive, he will change his 
confidence. The computation will * be done again. It dan'be done as many times js the 
student wishes untiUd^p is satisfit^d his potential number of points. He will then 
^sipial his completion by proper key press, and the appropriate number of pomts will be 
credited to hin\, ' ' ^ ^ 

The student will receive a feedback message on the display if his confidence is not 
one of the 11 characters named above. 

For multiple-choice questions, the student will distribute his confidence over all 
alternatives (as in COEOLl). His confidence must add to T -(ten) and he must strike a 
character f6r each alternative. Once again, the weight of the question supplied by the. 
^^or will be multiplied by the three-place decimal associated with the student's 
T^onse, TJiese products will be, rolled onto the display in payoff fields next to each 
alternative. If the student is satisfied, he just presses the appropriate key.- If \fe is npt 
satisfied, he will change his confidences until he is happy with the' payoff involved. When 
it is Tound that the student- has pressed his key without changing his confidence' 
assignments, he will be awarded the number of points he has assigned next to the 
correct alternative, , s 

For example^ if a question is worth 50 points, and the student distribi 



confidence as 
correct one, he 




ws on a four-alternative question whose srecond alternative is" 
receive 33 points: 



A. 
B. 
C. 



Payoff 

33 
33 
44 




Alternative 
Alternative 
Alternative 
Alternative 



Here too, should the student type a character into his confidence that is not one of 
the 11 characters mentioned, or if his conffdences do not add to 40, ^e will receive a 
feedback requesting him to correct his error in assigning, biS confidence to each of 
the alternatives, 

In sum, it is felt that a combination of reality testing, confidence measuring, and 
conceptual responding would form a useful baseline of a next generation instructional 
strategy (I DM). Research on refinement of decision rules using these dimensions should 
aid-^ther development of useful computer-based instructional materials in meaningful 
tutorial environments, ^ 



IMPLICATION^ OF EXPERIMENTAL FINDINGS 

Interpretation of the data can best he presented in tenuis of guidelines for pre- 
scribing instructional management, 

1 ■. ■ . >A 
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With respect to option control by students, the data suggest that we: 

(1) Establish high and low performer initial predictions for sets of similar 
instructional tasks. " 

(2) Provide 4iiaximal student-control for the predicted high performers. 
^ (3) Design maximal system cont/ol for predicted low performers. 

V (4) Track the performance ot all. 
(5) Adjust the degree of stadent-control, based upon the changes in per- 
formance according to pi empirical model that maintains the maximum 
percent of high performers. This model would have to ^be derived for each 
Subject-matter application since tasks and instrucjticuial materials would 
have unique complexities and forms. 
Related to the above is the ability to be able to discriminate, using some entry test 
battery, the high and low performers on an initial basis. To the degree that this can he 
done, the de^ignat^d student or system control options will be more or less appropriate. 

As noted from the above ECT analyses in the current study with a primarily verbal 
course like C0B0L2, the single best predictors were verbal entry tests such as the 
Primary Mental Abilities Test used for programmer aptitude selection, as well as the 
Cooperative English and, in another' case, a uniquely suited^ Programmer Aptitude test 
called the ATPP. The significance of the ATPP, as well as the other structure of intellect 
tests which showed up as significant in our discriminant analysis, emphasizes that there 
will be other unique charact^istics of any given instruction which^^vill also aid in 
discriminating the predicted high from the low performers. .These factors would have to 
be discerned from a structural analysis of the subject matter and its related tasks. (In the 
current instance, we are still doing analysis of this subject matter* by factor structure 
using multiple raters to arrive at a reliable index, vis-a^vis the Guilford Structure of 
Intellect characteristics.) 

Again, from the current study, supportive evidence for differential task transfer and 
the contribution of unique tasR and subject-matter characteristics comes from the com- 
parison of Division A and Division B predictors from the ECT batteries. For example, in/ 
the • introduc'tory^ part of the course, division A, the discriminant function ^gtf ts 
'characterized by the most general and smallest number x)f predictor tests. Wherel^n 
Division B, which was more heavily loaded with uni^e characteristics of COBOL 
programming and specific technological tasks,"~there were more variables present, and we 
found that the characteristi«s of these » ECT predictors, were more unique to the 
progratnming and specific tasys related to factor structure. Specifically, we refer to the 
fact that the ATPP, a uniquely oriented programmer aptitude test, was a heavily weighted 
factor un^er Division B prediqtion. Mbreovej^ there was the appearance of structure of 
f;intellect factors involving associative mcm^ and logical reasoning. In like manner, the 
anxiety test, IPAT B^ was a predictor of ^performance In the iijtroductory part of the 
course but dropped out as people became more familiar, comfortable, and sophisticated^ 
in the COBOL programming tasks. 

Carrying this logic ifurther, it would have. vety interesting to test this 

hypothesis of specific task transfer with even more unique* and specific, sophisticated 
COBOL course materials like the other two divisions which were not available for our 
experimental subjects during the conduct of this research. It would be predicted that 
probably more specific factors like logical reasoning and figural adaptive flexibility (the 
ability to change set \Vith new materials) would! take on4even greater importance, all 
relative of course to the specifc nature of the task the individual would be encountering. 

Combining the abqye noted discussion of op'jtion availability and relevant ECT for 
prediction of hid||pr low performers with the prc\)ious description of the self-assessment 
results provides an indication of a workable instructional dcvit.ion model to be tested in 
future research. It wouJLcl t^kf\. something like the following form. 
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Given that the high performer predicted by specific ent^ tests is more efiscient in 
his use of options and is more realistic about his ovm perform^ce, the self-assessment via 
^he.use of LOA could be used as a tratking device ^or technique for adjusting the degree 
>^ student control over the available remediational or accelerating .options within the 
course of instruction. When a predicted low performer, for example, begins to fall within 
the realistic range of predicted high^performers, that individual would th6n be ^owed 
more control over the use of available options. , ,\ 

Prom the pther side of the coin, when an indffj^KfS falls outside the range of reality ' 
testing and is predicted t6 be a low performer, thetithe adjustment would take the form 
of eliminating student control over available optjons until such time as the individual 
performance logins to come more in line with reality and, indeed, tmtil the preyed 
performance jumps back up to what a high performer would Be. This model, however, 
does require continued research in order to verify its appropriatdiess to various \ 
applications of instructional tasks. f 

As an adji4gct to this mpd^^l, it may also be' relevant to add other parameters which 
describe in a more sensitive wa|*the high or low p^formgr's state of understands, fhis 
might be done by use of th|^ revised confidence measures discussed earlier under ^ 
self-assessment. It may well be |hat a previously desigh&ted ^igh performer who yields. !^ 
sonie ivirealistic estimates of performance in some novel inaterial might be signaling that* 
he is getting into^?ep water^^^ can no longer^ handle the instructional tasks required of 
him. In this ca^, suppleinent«j|the probing of "that individual's understanding by the use 
of confidence techniques n^K provide additional indiGfes for . the kind of specific 
remediation unique to Jiis ^cciBirements. All'd? the above awaits further verification in 
real-world instructional environment, similar to that used within the current staidy. 



FORMAL THEORETICAL MODEL FOR INDIVIDUALIZED l|\iSTRUCTl^;j ^ 

The formal theoretical research has concentrated %r{* the developmenL^of a 
mathematical model for individualized instruction. To this end we have con cent^ted on 
the first four components of Pask's (1969) five requisite iftgredients for a model 
for instruction: \ T ' ' * a 

X (1) A representation of^the sui5ject matter .to be taught. " ^ ^ 

(2) A representation of the. educational goal— both terminal behavior and 

cognitive components as appropriate. '^i 

' (3) A representation of the initial student-state or the siue of the student 
upon entering the system.^ ^5 

(4) A representation of the current student^state, <5l*^e state of the student at 
any time. ' - ^ ^ ' % ^ \ 

(5) A representation of the teacliing system, including its teaching strategies. 
\ The model "deve^g^ed for individualized instruction is at present primarily 

descriptive, with extengi<^s' under- way to prescriptive guidelines for developing an 
individualized instructional environment. The work to date can be summarized under 
four headings: %i 

(1) Structuring Subject Matter 

(2) Representing Student States ♦ 

(3) Representing the Goal of Instruction 

(4) Identifying Paths Through the Subject Matter 

Consider the first problem: Structuring Subject Matter. Historically, attempts to 
chamctcrize subject-matter structures seem to fall into two classes. On th^ one hand, 
researchers such as Gape (e.g., 1965, 1969) have focused on task structures. Gagne's 
formula, *mat must one be able to do before. . leads to structures that focus on the 
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tasks a^^tudent must learn how 'to perform. Researchers such as Pask(1969), 
Guilfordtlp67), and Gagne(1971), on.th^ other hand, have focu^^e^on the content of 
subject piaster. Pask (1969) structures subject mat£er .along ^togJ^^rrf^ fomiula: 
*'What musti a person know before^ he can learn. . .y^^^ ^ 

Thus, kn adequate model for subject-matter structujr^must 'include both task- and 
content-oriented components. Prescriptively, our mod^ asserts that before specifying 
tasks, behavioral objective^ must be known. Before specifying subject-matter content, the 
body ^f knowledge to 'be studied must be defined. Neither of these steps can be made in 
a vacuum. The context in whi ch instnlction is to be o^fe^ed^piust be specified clearly 
before either ' behavioral objectives or the detanm^f the body of knowledge can be 
specified. The model asserts that the target population determines the context Ipx the 
instruction. Furthermore, the general body^ of knowledge mediated by, the context of 
instruction yields the requisite detailed description of the body of knowledge. Space 
prohibits more than this ver>- brief summary of the prescriptive aspects of the model for 
subject-matter structure. 

Descriptively, the model yields two fj^mary components: Tl^ Task in Context 
Structure (JICS) and the General ^Cognitive Net (GCN) for the'' context-conditioned 
communicable body of knowledge. Together, these components and their relationships 
make up^ the subject-matter structure. Both the TICS and the GCN are graphs. 
Graphically, they can be represented as a set of nodes ^vith connecting, directed arcs. The 
term gr^pfi is^used to indicate that there are no restrictions concerning the number of 
in-directed and out-directed arcs from any nod§ such as would be imposed by the 
. mathematical (Concept of a function. The descriptive aspects of the model for structuring 
•subject matter are included in a separate, report (see Appendix A). The document includes 
a statement of the j:heoretical axioms for the GCN and TICS, devejo£men,t of the notion 
of dependency, and a comprehensive application of the mpd^T^^ v 

The last tMree theoretical topics: Representing Student States, Representing the Goal 
of Instruction, and Identifying 'Faiths through the Subject Mailer will be dealt with in a 
later paper (in preparation). Briefly, the methodology of Inductive Logic (Carnap,' etc.) 
has been adopted to achieve a representation of initial and current states.- A set theoretic ; 
representation of the > goal of instruction is then developedi^om "the student state' 
descriptors. Finally, the notion^ of dependency developed within the Subject-matter 
structure portion of^the modef is applied to eliminate ^some student states as being 
impossible. The dependency relation then can be'shown to constrain the sequence of the 
remaining possible student states which the individual^n-pr-ogress throughT. ^Finally, an 
algorfthm is developed that leads to the identification of all posaiblg^ paths (successive 
student states) through the subject matter. " . . 
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COUr(Sl DEVELOmENT AND IMPLEMENTATION 



HISTORY OF COBOL DEVELOPMENT 

CdBOLl ^vas administered first to higii sciiool students for debugging and^ then to ^ 
Army students for evaluation. Data were collected and analyzed on a total of 
42 students. The evaluative data showed tKtL^ubsets of instruction needed to be smaller, 
for both learning and research purposes. .Thus arose the modular^ concept employed in, 
C0BOL2, the second iteration COBOL course developed by HumRRO. Also, more" 
opportunities for progf^m ^vriting and training in debugging procedures (through gaming) ^ 
were proyidefd for the student. - 

The IDM and associated support software also Underwent changes at this point, with 
a major element being the separation of the Jogic of the IDM from the instruction. The 
ramifications of this development were enormous because the IDM could be dynamic and . 
unconstrained by^the course logic. 

The C0B0L2 course and associated logic were debugged and evaluated with 83.. v ' 
students of various civilian and military backgrounds. Add to this tl\e 80 experimental * ^ 
subjects and HYe nuihber of students who have t^ken the COBOL materials totals to 205.» , 
During* the course, of 'running all these students and thecoijpomitant effort made to 
perfect the COBOL materials, a complete instryiiieft^T^stem evolved, including 
managerial, administrative, production, and evaluative tools and procedures. 

The topics addressed in the remainder of this sect^ion are: course design, author 
support, components of the coursevrare subsystem, production process, operation and 
administration, and revision process. , ^-^^^ 

COURSE DESIGN * ' ^ ^ 

Within the II)M the rules for pfresentatiqn of course material are defined and the 
software interface operationalizes Jhese rules. Since the whole instructional systenv is 
based on rules and the practical hardware/software limitations relevant to their execution, 
the cotirse author must structure his material so that it is compatrWe with both the rules, 
IDM, course structure, and limits ol-execution. , ' ' * 

Since the course isj^ase^^ a vvell-defined schiepxa of objectives, the first constraint 
on an Author is tttat his rnaterial must b^ structured so that it fits the design of the y 
^cpurse objeciives-that is, that there^is a singular terminal behavior defined by the single f 

cQurse objective, and the question objectives are single enabling objectives. The module 
. and division objectives are of an intermediate type, whereby' 'they are enabling to 
^whatever is above them (see Figure 6) and terminaj to whatever is below them. Because 
the mrodule o^pjective is the lowest-order terminal objective, it becomes the logical unit of 
instruction fo^ an author. 

The course structure (Figure' 7) reflects the essential elements of the objective 
schema. The set of behavioral objectives for a Division is sufficiently large and complex , 
to require further partitioning from subsets to sub-subsets. This produces a Module. ' 





Hierarchy of Objectives 
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Where: CO = Course objective 

DO = Division objective 

^ MO = Modufe objective 

QO Question objective 



Figure 6 



Modules are partitioned into sections. The first is the A (administrative) sectio/i, 
which contains a variety of administrative documentation that the student does not see, 
for the most part,. but which is used ^to provide course management information, Secorid 
is the T (telling) section in which relevant subject-matter inforn^ation is presented t<?^e, 
student. Third is the P (practice) section which permits the student to practice objective 
related behavior. The P-secUon^is^Joll^^ by the Q iquiz) section which tests for 
achievement of t^eJjehavi^aC^fBje^ A module can have several versions of each 

seojtion. The liin^ation is, of course, that each version covers the S£(me basic material and 
therefore t^ches to the same objective. The differences between versions are differences 
of tdtm afi^/eHiriiTlar^ather than content, ^or example: Version 1 of a T-section (Tj) 
may require extensive reading, whereas version 2 of the T-section (T2) for the same 
module may be high pictorial or contain smaller chunks of information per display. 

An author must write a module so that it instructs to one objective as that objective 
is represented in the prerequ^ite structure (Figure 8). That is, if a module (objective ^ 
does not have a linearly^r^reguisite module, the module being written* must be inde- 
pendent of other rnodules. .For example, in Division A, as represented in Figured, notice 
that modules F, G and H must each be written in a manner such that the author of any 
of these modules assumes only mastery of the objective "taught by Module E when the 
student is taking Modules F or G or H. The author of Module I, however, can assume 
that the student has mastery on the objectives taught in Modules F and G and H. The 
prerequisite structure Shown was established for the COB0L2 course daring its behavioral 
^design phase and is based on an analysis of ^thg^jfeiStive nestings of behavioral objectives 
within others. * 

^ The irAplic^tions of this design ^for the way a course author musT&aate his material 
is evident in Figure 9, vvliich is a partial diagram of the path structure through Division B 
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V Partial Pajh Diagram for Division B 
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Where: 

Moduiei Within the 
brackets Q J appear 
to^iether on a, menu 
for^t he student. 
In program paths *--&,' 
* J exit points 
&= entrance points 
* (a single rriodule 
or a menu). 
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(if the student chooses Module C "at^ his first choice point). Keeping in mind that the 
CDB0L2^ course^ is a-.self-cor^ined tutorial (no non-system instructional support) the 
authof^ni^st"^ create the instruction so that all students, no matter what path they took to 
get to^e module, can coniprehend and thereby master any module in the Division. 

* This structure was an' 'outgrowth of the IDM 'design permitting learner control over 
his path through the. subject matter. The other learner control features of the IDM had 
different effects on the authoring of modules. T-sections are witten so that general 
concepts are introduced first and, as the 'student goes on, the specifics are explained in 
detail. This technique enables the student to discern, as early as possible, whether he 
think^ he has sufficient mastery to skip to either the P- Or Q-section. It also enables the 
stud/nt who is in ,REVIEW or RECAP mode to find the information he needs as e^ly in 
th^Onodule as possible'. • 

f The structure of a P-section should resen^ble that of the T-Section in that exerdses 
aealing with the general. concepts should be first, followed by questions on specific 
enabling objectives' (for ^ the module). The P-section provides diill and, practice, and 
detailed explanation (in the 'form of associative displays) to complement and expand on 
the information presentea in the T-section, and should also provide content and format 
preparation for the Q.uiz. 

The Q-section must be a discrete, self-contained section including explicit directions 
to the student on the meghanics of answering the question(s) •that test the module 
objective.* It i§ important that the quiz be an accurate t^t of the objective and not be 
affected by the idiosyncracies of question presentation or methods of student response. 

Authors 'found the course design and structure to be beneficial during the creation 
of materials; because it imposed enough constraints on the instruction to malce different 
modules, written by ditfcrcnt authors, compatible in form and general order of content 
presentation, it allowed, hcwevgr, much individual frc^dom^to authors in the ereati)!JH||pf 
ivilhin-module stratejjies. This is exemplified by the fact that the modules which used 
gaming as an instructional method m(!i the IDM ^nd course structure requirements, as di4 
'the more conventional instructional techniques employed in some of the otl^er modules. 



AUTHOR SUPPORT 



\CAI ^s developed by t^ project (IMPACT and GRANT GJ.774) was characterized 
by separaJeTtwage.jjad-'f^rieval of Instructional content and instructional logic. This 
innovative technique, proven successful under operational conditions, gave rise to unique 
on-line authoring aids. 'One of , thdse, being ^ble to retrieve and display a pi^ce of text 
throughout the course whUe, storing Qwly a single copy in th v instructional text file§, 
resulte^ , in ^ very economical use of. file space. Economy in the form of inc^based 
^^cost-e^ectiveness was the result of allowing several people to work on the same piece of 
" instriJttion (module, division) simultaneously, thereby- amortizing coftiputer costs Over 
mbre useig and reducing total time on-line needed to create a given segment of 
instruction. * > ' 
^ — ^ Part ..of the time-sharing monitor and data-base management 'executive subsystem, 
called ZEUS, was, a simple author language called EDITOR. EDITOR could l?e mastered 
^ jquickly since the author need know only five commands to manipulate' the text files 
adroitly. ' (See Section 5, Software Support). The^e five commands are - CREATE, 
MODIFY^ DELETE, COPY, and DISPLAY. Their meanings arc solf-cxplanatory. All an 
Author need do is enter a command ancT an^ object file location idontifirr, and the next 
thing he s^ is the requested file element; In all cases except DISPLAY^ the element is 
automatically updated, according to the command is'sut'd, aCTor the author si[{nals that he 
^ is finished with the element. , * - - 



■ The separation of text and logic; in combination with EDITOR, made possible the 
developrKient of an even more sophisticated and powerful authoring aid-SUPEREDITOR. 
The role of SUPEREDITOg^ in the courseware subsystem is to facilitate the manipulation 
of the text files for naive, as well as sophisticated, authors and editors when either 
authoring or editing is done ihteractively, on-line. To use SUPEREDITOR an author nee'd" 
only know the operational dfefmitions of the SUPEREDITOR modes as shown in Table 8 ^ 
Unlike EDITOR, SUPEREDITOR allows the author to work on a string of disk elements 
and he has some new capabilities which make string m'knipulation even easier. For 
exaihple, PRE-FORMAT mode allows the author to select an existing fil^e element (or 
create a new one) to .function as a template. Thereby the new. element being created 
already has a structure when it appears on-line and the author need only fill in the 
content. This feature is especially useful for standardized pages such as documentation 
pages. ■ ^ 

• The bulk of the course-specific docunientation is resident on tho instructional text 
files. For the most part, the student does not see this documentation, but it can be 
accessed by course authors for modification anc}/or°yiewhg in the same manner as any 
6ther element. It is not always desirable, in terms ^'timc and ccst-effectiveness, to work 
with and inspect either instructional or docunientation iteir." oij-line. Therefore, a 
hard-copy, off-line reproduction of Mie instructional text files vvaG implemented via the 
File Activity .Control System (FACS). (See Section 5, Software, and Appendix" F.) 
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SUPEREDITOR Modes' 


Symbol 


/ Mode 


operation 


0 


ANTIDOTE * 


*■ > " 
Retrieves a deleted elennent if no other operation h^s mterveaed between 

'DELETE and ANTIDOTE. *"* 


_ 1 • 


DEL^f^ 


Inactivates ^ disk location. . ^ ** >^ ♦ 


2 • 


DISPLAY 


Retrieves the cpntents of a disk locatidn on*the CRT screen for viewing, but-, 
does not pernni4 any permanent altoratron of the element. 


3 


CREATE 


Activates a disk location arrd permits the User to store te^t on the elemeht. 


, 4 


MODIFY 


Accesses an active'disk location and per.Tiits the tic^r tc change tho content - , 
and^or structure of the resident element. 5* 


5 


PRE FORMAT 


Copies a previously stored template into a newly dctivateddisk location 
and permits the user to 'input contentlnto the s'l ucture. 


6 
Ik 


DISK DISPLAY. 


J^etrieyes the dFsk location number of an element bemg viewed via DISPLAY 
and places it in the uppei left corner o^ the Jornent. ' ' ' 


7 -EDITOR 


Allows the user to write his? own Editor cornmands, two at a tirtie. Adds these 
to the Editor page/. / • ^ * • 


8 


SIGN OFF ' 


Allows the user tp exit from SUPRED 


9 


DISK TITLES* 


Exet:utcs tWeMist command which shows the label of all specified active disk 
.locations ;3nd the n<ime of the last pcson wh ) worked on each. 


^ A 


JOB ENTRY •''Allows the user to retrieve and execute any progrann in thr^ D f ilo. 


B 

0 


TITLE MODIFY Allows the user to nnakc changes in thu Idbcl of a-disk locdtion And to change 
* the^content and/or structure of.the oJrmcnt, if dosirrd. - - 

r ^ ! 
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Separate TITLE listings ar^ also produced by FACS, It contains such information as 
the entry's location on the text file, its identifier, whether it, is seen .by t^ie student, ^nd 
what tjjpe of student response -it ,c£dls for. An author can follow^ the flb)v of a module 
just by perusing the TITLE listing. This check also insures that the module flow does not 
violate the IDM rules for module structure. 4^ ' . 

In order for the IDM to make decisions based on quiz scores, the scores have to be 
standardized, but different criterion tasks are not al>vjays aiq:ienable to' similar ^coring. 
Therefore, a speciaj computation element. Figure^ id, was ^developed to help authors 
assfgn raw- scores to quiz questions/tasks^ and then transform i\\em to standardized IDM 
scores. The FACS , copy of the computation element is. a useful tool for the author to 
have \vhen he is debugging a quiz. ' - ' " 

During course administratron and cfebu^, the FACS ^cc^ies of the standardized 
documenteftion pages (especially the A-section) enable aiy ^author or administrator to 
understand what should happen during the instruction, even if he is unfamiliar with the 
module as a whole. All authors must provide documentation with any module they 
^create, and thi^ practice has proved to be beneficial both when the material is being 
prepared for on-line operation and after 'it is operational.' 

Overall, the author support facilities have enabled' authors with little .or no 
gramming skill tp dreate and put bn-line^ effective textual materials for CAI. 

I 

COMPOIVrENTS OF THE COURSEWARE SUBSYSTEM o . • - . , 

. • - * 

* The set of instructional materiah^ for computerized training by this project consists , 

• of the following products: . ' . " . ^ . ^ 

o Text* . ^ \ , . ^ 

, ' o Auxiliary^ Visuals* * * 5 - ' ^ 

o Glossary' . ' • ^ 

o Student .Reference Manual - ' , ' ' • 

* ' The t^xt was developed for on:line, interactive application in a total tutorial CAI 
mode. The* text" files contain both the instructional materials for student viewing anc^ 
on-line documentation. * . ' , * ^ 

Two main courses were ^evelgped. The introduction to CAI Course (INTRO) is used 
to introduce the student to the features of the CAI environment. It is independent of 
any subsequent substantive course and serves only to famijiari":e the student with the 
parameters and mechanics ,of .the CAl system in which he will learn. Ther^ are 13^ 
modules in the -INTRO course, and it takes the students from two^ to five hours to 
cbmplete all of them. An alternate version of INTRO was developed whiclrFeplaces nine 
pf the modules with an off-line leetilre and retains on-line only the mcpules covering 

• keyboard practice and IDM features. Completion time for this version is a nfaximum of 
, three hours. ' - ^ , 

The C0B0L2 course teaches the student some of the basic concepts of data 
processing using the CQmmon Business Oriented Language (COBOL) as a vehicle. The 
course consists of four ^divisions which <jontain 33 "module.s with an estimated average 
completion time of 45 hour?. By '^cutting and. pasting" the FACS representation of the 
on-line text, a CMI (quizzes and j)rescription remained on-line) version of portions of the 
COBOL2 course was developed. The CMI students did not do qiiit»e as well as their CAI ' 
^ counterparts in 'terms of transit time and performance -but the results were still accept- 
* able. The differences could be explained .by realizing that little attempt was made to 
- — — . — jada^LLl^^ off-line materials for the rlew media. 

Available in microfiche form. ' 4 c\ " ' " 

^Available in either «^5mm or 16mm^^ ' 4 O r ^ 
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Because "of theMack of an 'efficient graphics capability (simple graphics could be 
done jasing yphanumeric and special characters) on the CRT, an auxiliary visual device 
was includecj in the student terminal configuration. This device was a' random-access, 
Hmm projector whiclfl^s entirely under computer control (author specified). The device 
wa^Ote^d when color, graphics, animation, or illustration was required^ ol: if the CRT was 
already in use and the author wanted to present additional or supplemental text, or if an 
author- wanted a dynamic pictorial presentation of the subject matter rather than a 
dynamic semantic presentation (via^the CRT). 

The glossary or inquiry capability was presented to the student as an IDM feature. 
For the most part the student used it as he, would a dictionary, but in the "debugging" 
modules he inquired as to the meaning of various COBOL diagaostic codes for program 
error messages, using the srime technique. Terms v^rith their .definitions were input mto 
the glossary file by one* of two methods: (a) an author, when creating material, specified 
any terms that should be included; (bj if three different students requested a term, its 
definition v^as input. r ^ " ' 

The student manual was organized, like a standard reference book, rathet than by 
course structure. This Avas done to familiarize the student with using a programmer's 
manual, since programming normally requires considerable facility in ?he use ot reference 
manuals. Since the student has the manual with him almost all the time, the author must 
be careful that he does not negate quiz results either in a current module or a future one, 
by putting? too much cfourse-specific information in the manual. 

The courseware components then 'represented a diverse set of hiedia and capabilities 
^-available to an' author, so that each could be employed optimally in the CAPleaming 



^environpient. 



PRODUCTIOIM PPbsESS 

The approach taken to the development of instructional materials has been to use a 
lyiultidisciplinary team to design;,, develop, .andMmplement course materials. The team * 
capabilities include instructional design, subject-matter expertise, instructional pro- 
^ gramming, editorial skills, hardware/software expertise, 'media selection and utilization, 
and production and management skills. ^ • ' ^ 

The instructional designer establishes behavioral requirements, encodes them into 
instructional objectives, and designs the IDM that will best serve the needs of th^ " 
. students and the subject matter. The subject-matter expert provides ''raw" content to the 
instructional designer and instructional programmer and serves as a consultant when the 
programmer structures the content for implementation in a CAI mode. The instructional 
programmer is a specialist in lasing computers'effectively and efficiently for instruction. 

The editor reviews the draft instruction, which is usually delivered by the instruc- 
tional pro^ammer, on a paper representation of the CRT screen containing all the • 
, information for a particular element— text, chatacteristics (to be entered in the title 
listing^)*, and so forth. The^ editor is examining the material for accuracy, consistency 
(inter; and intra-modular), and style. Auxiliary material (visuals, manual, glossary) is 
revievved^along ^th the future on-line materials, at this time. 

The material next^ ^es to the fedijiicians who input the instructional text and 
program the instructional logic. ^^"''^^ 

After the material is implemented and before studefnts are allowed to take it, the 
course must go through a debugging phase. Material moves do\^Ti tlir^production chain 
modularly and the first debug is done a module at a time. The dohug includes on-line 
' ' chocks for proper linkage among text elements, sami;)linps of all possible ansv/ei-s to 
^ , questions to msure proper analysis^ and branchinf^, vrification of quiz scoring usmg all 
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possible permutations- of carre6t, and partially, corre^ct answers, and an off-line check, 

ysing FACS output, for textual errors* (^pelliag; syntax, etc.). After this step is satis- 
^; factorily completed,, several naive students go through the material to further '*shake 

down" any bugs. _ . 

Only when the entire production process,' from design to on-line test and debugging, 
.^^ js 'satisfactorily completed is the crurse deemed ready for students to see in* a full 

operational mode. . 

OPeRATlOW AMD ADMINISTRATION 

After ther INTRO and pOBOL2 were thoroughly debugged, students were permitted 
to take the courses. Before the students went on-line, they had^ to take the Entry 
Characteristics Test (ECT) battery. At this time they were given an overview of the 
projecj, told the function of the ECT, and given an explanation of what they could' 
, expect to gain from taking the COBOL course. ECT lasted for three half days and 
screening took place afto.r the ,first session. COB0L2 was written *for students who 
exhibited a mininium aptitude (equivalent to a low Ggracje) on the programmer aptitude 
test, the ATPP, and this was the test upon which students were screened. 

To take the course proper, as part <X the first instructional session the student 
keypunches spme data cards and is taken to* the; compyfer center to see the program run, 
^ using the cards he has just punched. The student then enters his carrel and starts the 
INTRO ^course! From here on, the stud«?nt§ are scheduled for sessions of three hours a ~ 
day, five dtty^ a week until they are finished with the course. They arrive in the 
beginning of a session, pick up their folders containing'supplies from the proctor, and 
spend the remainder of the session in their carrels. If they perceive problems during the 
instruction, they call the proctor to their carrel. ^ 

The proctor is part of the administrative staff responsible for the operation oi^ the 
course. The members of the administrative staff are: 

The Director for Daily Operations^ who is responsible for efficient day-to-day 
"running" of the coiurse. He ser\^es as liaison to the/directors for other components of the 
total system. It i§ his responsibility to insure t/iat required system {kny component) 
changes are accomplished in sUch a manner that student progress is minimally affected. In 
effect, he is a traffic manager for problem solutions and computer utilizatiqn while 
students are on-line. Flis duties also include supervision of the proctoring staff. 

The Operations Monitor (OM), who js responsible for -the smooth running of a 
particular student period/shift and supcrvi^ng the proctors on that shift, lie serves as a 
resource for the proctors and is* the only person assigned to the shift who can give the ' 
student relevan^suBject-matter information. ' 

Tha^^roc^or, whose function Ms pur^ely procedural. The students should be 
informed of this.' The proctor is responsible for insuring thaTTRe student has all required 
materials, correcting minor hardware diffiQuItfe. {auxiliary vi^al display does not align 
properly witih primary display, etc.), answering sVstem-gener^ted proctor calls, diagnosing 
a student'sproblem, and, if it is subject-matter specific, referring the problem to the OM. • 
. ^ Mosb^ proctor calls are student initiated, and the proctor goes to the student, 
•di^gju^s^ the student's problem, and effects a solution. In our tutorial CAI application it 
is crucial that the course administrator and/or evaluator l)e-al)le to determine what 
transpired during the proctor-stucient interaction. Therefore, a proctor trouble "report 
(TR) was^ desif^ned and developed to document all proctor calls, both student and system 
generated (Figure 11). The TR was part of the VOYEUR course. VOYEUR was created 
as a -cQurse, although it was purely for* administrative purj^o^cs. so that the 



42 



ERIC 



^ • • . 46 




proctor-generated recoftis were compatible with all student records, so as to allow 
standardized manipulation and analysis. 

Once a day the data in the TR file are reproduced in,, hard copy and are 
disseminated to an appropriate ref^riT who takes action to eliminate the cause that 
generated the proctor call. If the i^feree feels that the problem was unique and does not 
have evidence that it could reoccur for other students, he need not take any action. The 
TR record is updated after referee action takes place. At the' end of a student group, all 
non-upSated records are. identified and the Director for Daily Operations insures that 
• some referee action is t^ken. These procedures provide for the quality control of the 
instruction while in an operational mode. 

During course administration the off-line* author support facilities (FACS, TITLE 
listings, Ptc.) are used for administrator support. Because of the extensive documentation 
in the FACS, proctors' and OMs can accu]{^tely analyze and salve most student problems 
without referring back to the original author or having to know how to -read the 
instructional logic printouts. * 

Course operation functions as if it were in an applied operational setting.. It is an 
independent system administered by para-professionals which wo^ld, in a.scUc^ol setting 
for example, free the professional subject-matter expert from the relatively minial tasks 
associated with course administration. 

« 

REVISION PROCESS ' 

The COBOL2 course was evaluated for revision requiremerits after each group of 
students completed the course. The revision and' review process drew upon computer- 
generated data management reports (daily and module summaries. Table 9f, TRs, and ^e 
expertise of behavioral scientists, instructional technologists, subject matter experts, and 
software support staff. . ^ 

A systematic methodology for diagnbsmg revision requirements was developed. It is 
being written up now as part of a paper on formative evaluation. When a ^'repair" has to 
be- made, the new material goes back through the production process, entering at an 
appropriate point and going through a debug phase before it becomes part of tlie course. 
Application of this algorithm* led to no modules being rejected by the time the experi- 
mental subjects took the C0B0L2 course. 



Table 9 



Summary of Quality Control Pwgrams for CAl Course 



Pr09ram 



^ Output 



Acceptance/Rejection 
Student attendance 

t 

Student comments ^ 

Student Glossary Request Found 

'Student Glossary Request Not Found 

Programming Errors 

Programmmg Errors II 

Performance Summary 
(one table/objective) 

Summary Student Response Matrix 
(1 matrix/instructional unit) 



Student Opinions 
Student Options 

Student comments 
Student Response Llstmg 

Answers for Quiz Attempts 

Most Frequently Incorrect Answars 



Most Frequently Correct Answers 

Questions with Longest Cun\ulative 
Response Time ^ 



Flags substandard instruction units by statistical criteria, as soon 
, as»possible, so revision can be initiated. 

Reports attendance record, (daily and cumulating), time of signon 
and signoff for the day per student, furthest 'piace in course, and 
i^mulative time spent on course. 

Reports student opinion ratings and all-comments referenced by 
^he student number and the place where it occurred. ' 

Reports word requested, stucient number, location at time of 

request, data and time request was made. 

Same as above. ^ 

Questions that were incorrectly graded. 

Records inconsistencies in data compiled for specific items. 

Lists raw and transformed quiz score for each student and for 
each attempt. * ' \ , 

Each row of th^ matrix represents a question. Each column 
represents a student. The cells of the matrix indicate correct 
answers, incorrect answers, and skipped questions. Marginal 
percenjages show the percent correct by student and by question. 
Prints a value from 0 to 9 for each studer^t. 

Indicates number and ^ype of student control options exercised 

by each student, and wlFiere they were invoked. ^ 

ft 

Verbatim, with student and course location identifier. 

For each question (except quizzes) lists actual response typed by 
student, as well as certain counters and switches. 

Lists quiz responses by student. Dj^ftinguishes fir^t, second, and 
subsequent attempts. ^ i ^ 

Lists the five questions in the instructional unit with the highest 
frequency of incorrect answers. For each question,- it prints: 
number of attempts; number of incorrect answers; number of 
attempts with response greater tharj 10 minutes; number oi^ 
unanticipatecl responses; number of answers with spelling margin, 
punctuation, or subject matter spec f ic errors; and the cumulative 
time in seconds spent answering the question. 

Provides same information as above. 

Same as above. ' I ^ * 
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Section 5 



SQFTWARE SUPPORT 



Software research and development has resulted in the development of a com- 
prehensive, operational softwja/e subsystem. that performs all software tasks requiredin a 
computer-a4rrupistered instruction environment. Appendix G provides.an overview ^dfllip 
software subs^tem. i - ' - 

The software subsystem can be divided Jnto real-time and off-line components. 
Real-time subsystem components include the f611o\ving. 



OS/3700 
Zeus 



Coursewriter III 



IBM 370 Operating System 

The on-line* monitor developed at HumRRO used to 
interface student terminals \Vith Coursewriter. Zeus 
includes three distinct*subsystems. 

Editor: for interactive authoring developrnent. 

Director; for locating^'and retrieving disk stored 
information. 

RJE: for inputting, initiating and executing jobs 
Temotely. ^ ' 

IBM^s Coursewriter CAI software subsystem with 
two important extensions. * - 

ft/ 

Interface: maintains on-line updated ^description 
of student attributes, coursfe component pre- — 
requisites, and in general controls all intermodule 
branching as well as some intramodulfe branching. 

Functions: spe^cially develoj>ed functions to^xtehd 
Coursewriter response analysis, data reporcTing, 
branching, etc., capabilities. 





Figure, 12 summarizes this on-line software subsystem anpl^so depicts terminals, random 
ac(;ess storage, and printed output. ^ * 4 

Of particular importance in this subsystem is th^Hnterface component The Interface 
assumes that a CAI course is structured in a particular manner. A course is composed of 
one or more divisions with each division having erne or more modules. A module contains 
a Telling (T), a Practice (P), and a Question (O) sectipn. One of the primary features of 
the Interface is that it permits an author to design an individual course for each student 
with respect to^di^ision and module definitions and structure, that Is. each student can 
have his own course structured for him by an autlvor, using the pool of available modules 
and module components (T, P, Q sections). Jnterfoce permits the author to establish 
individual module prerequisites for each student and controls intermodule and |ntra- 
module transfers for each student^ 'Appendix H provides\^.ore details concerning Interface 
logic, operation, and use. 



From this brief description, it can be seen that authoring activity consists ot two 
distinct operations. On the one hand, a pool of T, P, Q sections must be developed for 
ctuc'ont use. Given the pool of T^P, Q sections, the author must next structure them into 
a course for each st\ident individually. This entails specifying the individual's course 
div^cions, division modules, and each module's structure. Incidentally; the author can 
specify a different module structure with;, if desired, different T, P, and Q sections' for 
successive encounters mth the module by the'student. In addition the author specifies 
through Interface prerequisites for the individual student for individual modules. 

Interface assembles th^se data into an on-line record used to control inter- and' 
intra^odule trans^fcrs. Specifically a student can exorcise several Interface bptionr' * 
including the follo>ving: c 

REVIEW Restart the current module at the beginning. ' ' * ^ ' 

PRACTlCli: y Move forward in t^c present module to the P section. 

Move forward in the present module to the section. \ 
RECAP Recapitulates module already completed (System 

^ presents a menu of completed modules for selection). ^ 

The Interface also schedules remediations and presents appropriate options after a module 
is ' ompletod (end ^)f 0 section). Thus, in on-line operations with the .student, the 
Ir -'rfrce is in complete control of branching and has an updated description of s^tudent 
at: .butcs. - % 

^ Cour?e\vriter extensions have been affected through the development of 12 Course- 
writer, functions. Tftc functions provide |he following capabilities. 

(1) feelpctive cleajring of Coursewriter III counters. . y 

J[2) Sc^cc[.ive>clearing of C^^writer ni swflches. ^ ' \^ ^---^ 

(3) Doteri^iining the length of messjiges in any CQursewtiter III buHbr. 

(4) Performing complex arithmetical opJerations. , * , 
' (5) Scanning a^uffer anci trar^fcrnng its nAimeric content to ^ counter. 

^ (G^ Selective loading of .multiple counters from buffers or auxiliary storage. 

(7) Selective placement of an EOB in a l^uffer. - ' ^ 

(8) Recording sdident data when fequired., 

,(9) Removing specified character strings from a buffer. ^ v . ( 

(10) Determining frequencies of occurrences of specified character strings in 
student responds. 

(11) SelecWe storing of miiltiple Coureewriter III counters in buffers or 
auxiliary storage. 

(12) Scanning student responses and rpcording data concerning items in the - 
response relative to the COBOL language ^requirements regarding pro-. 

J [trammer defined names, ]<?served words, an<^l literals (syntok). 
In particular, a sophisticated extcnj^on of Coursewriter has been achieved that permits^ 
the analysis of complex student responses. This" advance was ossential for th^ analysis of^ 
slurient input of COBOL syntactic structures. Appendix f provides the details concerning 
these Coursewriter extensions. M ' 

Off'line software components include the following: 

IDES A data storage, retrieval and analysis subsystem with two 
components. 

BMD: Biomedical Statistical analysis package. 

Storage/Retrieval: The dala stordge /retrieval compouonl> 

FACS File Activity Control System— an author aid that provides 
management control of and reports on authoring activity. 
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These off-line components are depicted in Figure 13 which, also depicts data input, 
program card* input, random access storage and report; printing. 

Of particular importance is the storage/retrieval aspect of IDES. Initially (lDES-1) 
IDES structured data in the form of lists. Th^ is, list processing techniques were used to 
store and retrieve data. This was acoomplished through an extended SLIP processor. 
Subsequent research indicated that, more standard data processing techpiques could be 
used to perform these functions. This led to the development of a more conventional 
storj^e and retxievAl version of IDES (IDES-2). This improved version of IDES is PO/l 

^ based. . r 

^Thus, IDES has eVolved into a comprehensive system capable of storing and 
retrieving student generated data. Coupled with the BMD, this allows for selective analysis 
pf data in an off-line environment for research and modeling purposes. Details of IDES-i ^ 

. are given in Appendix J and details of IDES-2 ai'e given in Appendix K. ^ 
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Section 6 * * 

PUBLICATIONS 

' -' . ^ ' ■ •, ' 

SOFTWARE DOCOMENTATION SERIES . . ' 

♦ 

Overview of the Cbmputer-Administered Instruction System. ^^y John Stelzer 
and Jean Garneau, Technfcal Repbrl''72-21, August 1972. . 

rhe IMPACT Data ^Evaluation System-Version 2 (IDES-2), by Leslie Willis 
and Jc^hn Stefz^r, Research* Product RP-Dl-72-1, August; 1972. ^ 

IIL *The IMP'ACTData Evaluation System-Version 1 (IDES-l ), by John Stelzer 
and Leslie Willis, Research Product RP-Dl;72-2, August, 1972. 

IV. The Interface^ Subsystem Framework for Instructional Decision Modeling, by^ 
William Underhill and' John Stelzer, Research Product RP-Dl-72-3 Aueust 

v.. File Activity Control System (FAVS), by Uslie Willis, Jean G^eau^and\^ - 
John Stelzer, Refsearch Product RP-51-72'4, August 1972. ^ * 



VL Volume 7, ZeusJ^nctlons and-Design Concepts, by Jean Gtoeau anfl John 
Stelzer and Votive 2, Zeus Program Logic Descriptions, by Jean Gameau, 
• William Underhilt and Doris Shuford, both Research Product RplDl-75-5 ' * 
August 1972. - , - ' 

VIL IMPACT'S Computer-Adniinistered Instruction Sopmre Subsystem, Course- 
miter HI and Its Functions; by Doris Shuford and John Stelzer, Research 
Product Rif-Dl-72-6, August 1972/ v ' / 

VIIL Computer-Adn^inist^re^ Instruction Computer Program Logic for C0B0L2 
Course of Instruction, by Douglas Spencer, Elizabeth Sowell, Leslie Willis 
and Jean Garneau, Research Product RP-Dl-72-7, August 11972. 

. Projet:t IMPACT-^omputer-Administered^nstruction: Functions for the 
Coursewriter III Language, Research By-Product; RBP-Dl-71-2, bv Project 
IMPACT Staff, June 1971.' , . » J' J 

P,RQFESSIONAL PAPERS ^ 

CQurse Modularization ApplJ^: The Interface System and Its Implications for 
Sequence Control and Datcmnalysis, by E. W. Schneider, Professional Paper 10-73 
- November 1973- (Also presented ,at,the meeting of the-Association of Instructional ' 
Systems (ADIS), Chicago, Illinois, Apr;l 1972. • ; 

^ mo Should Develop Instructional Mjiterials (6r-CAI? by "Robert J. Seidel, Professional 
^ Paper 20-71, October 1971. (Also presented 'lat the Computers in Instruction Con- 
ference, Los, Angeles, "California; October 1970.) • ' 

„••."' O . 



Theories and Strategies Related to Measurement An Individualized Instruction, Ijy 
Robert J. Seidel, Professional *'aper 2-71, March 1971. (AIsq presented at the 
American Psychological Association Convention, Miami Beach, Florida, September 
1970.) Published- in Educational Technology, September 1971. 



MISCEktAIMBDUS PRESENTATIONS* -* • . . 

"Towards Understanding the Value of* Learner Controlled Instructional Sequencing,'; 
presentation by Robert J. SeideLat Association for the Development of Computer- 
• Based Instructional Systems (ADCIS) August, 1973, Ann Arbor, Michigan. 



TECHNICAL REPORTS 



"Project IMPACT Courseware Subsystem: Vol. 1, Innovative Procedures for Develop- 
ment and Administration," by Michael'J. Hillelso'hn, Technical Report in preparation. 

, "An Axiomatic Theory -of Subject Matter Structure/' by Edward Kingsley and 
John Stel^er, Technical Report in preparation. i 
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Appendix A 

AN AXIOMATIC THEORY OF SUBJECT MATTtR STRUCTURE 

HumRRO Technical Report, by Edward H. Kingsley and John Stelzer (in press) • 

Summary 

An adequate theory of instruction must include as components at least the 
following: 

A representation of the subject matter 
. ^ A representation of the instructional goaf 

A representation of the initial and current student states 
A representation of the instructional strategies ; * 

This paper focuses on the first of these components: representation of subject matter. It 
is the hope of the authors that the rigorous descriptive theory that is developed will serve 
as a foundation on w\\\q\\ ^can be built theoretical extensions to include the remaining 
listed components. 

It must be stressed that at present the theory is purely descriptive. That is, it is not 
at present possible to derive very many prescriptions to be used in controlling instruc- 
tional branching, structuring material, etc. These results hopefully Wilfbe achievable when 
the descriptive theory is more complete. R^ardless, the development of tl\e descriptive 
theory is recognized as the first? step in an ultimately pragmatic and applicable thcoJry. 
The descriptive theory itself will result in a significant step forward with regard to clarity 
and precision in terms of conceptualizing about the instructional process, 

The theory as presented, in contradistinction topmost theories for structuring subject 
matter, divides subject matter into two distinct categories: Content* and Tasks. Assump- 
tions are introduced that lead to the structuring of the content portion via a net or graph 
^approach. A hierarchy of content elements is introduced leading to a hierarchical 
definition of the notion of dependency. Linear algebra methods ai^e employed in order to* 
derive methods used to determine and to display the complete^ dependency structure for 
subject-matter content. ' , ^ 

On -the task side, assumptions are introduced that lead to a structured task space 
underlying the content strucfiu^e. The behaviorally oriented tasks are then related to the 
content structure via coordinating relations. The notion of t;omprehension of content 
elements, is introduced and'. related t9 thQ task structure. 

A very detailed ^example of the complete theory is introduced and the theoretical 
concepts are discussed and describxid ^vithin the framework of the example. 
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Appendix B 



PROJECT IfAPAQ COURSEWARE SUBSYSTEM: VOLUME 1, 
INNOVATIVE PROCEDURES FOR 
DEVELOPrAENT AND AD^MINISTRATION 

HumBRO Technical Report by Michael J. Hillelsohn (in preparation) 



Summary 

Bajckground 

The objective of Project IMPACT was to evolve a series of prototype systems of 
Computer-Administered Instruction (CAI)-in order to produce a total CAI system that is 
effective, efficient, and cost-effective for operational use in a training/instructional 
environment. ' 

The total prototype system includes four main components: 

(1) Hard\vare-i;he computer, student stations, and related equipment, 
. (2) Software-^he computer programming systems that control operation of the 
hardware. ' 

(3) Courses of Ihstruction-the actual content and logic of courses administered 
by the computers. 

(4) Instructional Decision Models -(IDMs)-the rules and strategies by which 
specific course content is provided to an individual student. 

The way in which the content of instruction is prepared, stored in the computer, 
managed by the computer while students are taking a course, and managed off-line by 
course authors and administrators is the key to the efficiency and effectiveness of the 
total CAI system. All components of the system interact in the courseware subsystem, 
v/heie one visible product, a course of instruction, interfaces with the learner. Develop- 
ment of the instruction v^thin the constraints specified by'^the other three subsystems, 
and administration of the instruction so' that all facets of the total system are optimally 
employed for the ultimate benefit of the student are the subject of this report. ' 

Objectives ' " * 

The original IMPACT objective (in 1967) for developing a course of instruction was 
as a vehicle enabling validation of the components of the total CAI system. After the 
first iteration of the course was tested, a period of reassessment resulted in more 
stringent and exhaustive objectives for the courseware subsystem. 

Maintaining the course materials as a valid vehicle fbr instructional design remained 
the primary objective. Additionally, th'e subsystem had become so large fone course was 
lOver 30 hours long) that efficient management techniques W to be incorporated— boHi 
on-line and off-line— for the development and administrativ^vactivities related to the 
courseware subsystem. Increased cost-effectiveness in the production of CAI materials was 
another goal during the second iteration of the courseware subsystenv! 

Also of extreme' importance iVi the second iteration was the objective of flexibility 
of courseware. In individualized instruction, a toacher /author must be free to develop 
creative and adaptive materials to meet the student's needs as they vary over time 
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throughout a course of instruction. Flexibility also implies that the • courseware^ is 
amenable to change withput disrupting the entire course administration. 

Approach and Development * 

All of the objectives were met by developing an innovative set of standardized 
procedures for the design, production, and documentation of the IMPACT Courseware 
Subsystem. r 4 

The most important software development, using the computer as an efficient tool, 
\ya^ the physical and^ logical separation of instructional text from the instructional logic 
files. Thfe IMPACT innovation aided the attainment of the aforementioned objectives by 
permitting on-line creation and modification of text of logic independently. 

Thfe capability resulted in' effective file management, econoifiy of text storage, 
reduced on-line development time, efficient on- and ofMine retrieval and modification of 
each of the files, and more adaptive instructional decision modek (IDMs). OfMine 
developments, such as the 'File Activity Control System (FACS) and title listings, were 
also made possible and practical because of the text and logic sijparation. These products 
in turn made it feasible for course managers and authors/teachers to do -much of their 
work ofMine and thereby use relatively expensive, computer time most effectively and 
efficiently. - " . - 

Development of the subsystem was evolutionary and iterative. The requirements of 
potential users were' anticipated, and a multidisciplinar^ team de^signed and implemented 
solutions to these requirements. After implementation, a period of assessment occunred, 
during which the innovation was^ evaluated. If successful, the technique became part of 
the subsystem; if not, additional solutions were tested until a satisfactory one was found. 




ERJC 



\ 



59 

6'1 



Appendix C ' • ' 

^ COURSE MODULARIZATION APPLIED: ^ 
THE INTERFACE SYSTEM AND ITS IfAPLICATIONS FOR 
SEQUENCE CONTROL AND DATA ANALYSIS 

HumRRO Professional Paper 10-73, by E.W/ Schneider. IMovember 1973; 
Presented at the iVleeting of the Associatton for the Development of 
Instructional Systems (ADIS)/April 1972 • 

Abstract 0 

The Interface System is a comprehensive method for developing and managing CAI 
or CMI courses composed of sets of instructional modules. Each module is defined by 
one or more behavioral objectives, and by a list of prerequisite modules that must be 
completed successfully before the specific module can be attempted. The System's key 
components are (a) a standard general structure for all jnodul.es,_ (Jd)^ consistent jnethod 
of labeling- logie and text elements, and (c) computer programs (presently written in 
Coursewriter ivith Assembly Language functions) to regulate inter-module student traffic, 
and execute system-controlled, and student-controlled instructional decisions. • 
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^; Appendix D , 
.;1._C0WS1WARE 

» This appendix consists of the C0B0L2 and INTRO courses; ^<^ith ■afr supplementary 
matenals. The form m, which th^aterials are delivered is as follows: 

COB0L2 and INl>0^'xf -and Logic) - microfiche 

Glossary ' ' ' microfiche ' . 

StudentJKr^rence Manual . loose-leaf bound ' 

Au5iH% visuals__ - ' . I6mm film 



Appendix E 

LETTER ON DISSEMINATION OF COURSE MATERIALS 




THE UNIVERSITY OF TEXAS AT AUSTIN 

COMPUTER-ASSISTED INSTRUCTION LABORATORY 
AUSTIN, TEXAS 78712 

Sutton Hall 512 



April 19, 1972 



Dr. Robert Seidel ' • 

Senior Staff Scientist 

HumRRO 

300 North Washington St. 
Alexandria, Virginia 22314 

Dear Bob: 

f 

I have had a chance to 'review the .documentation and computer listings 
you and Ed Schneider gave me during my recent visit to HumRRO. I 
can report 'that my initial impressions during the visit were quite 
correct; this is indeedWery useful material. Already I have decided 
to incorporate several of your ideas and developments, which I know 
are based-ofl-sound* theory as well as applied experience, into tive-"' 
design of the TICCIT data management/ system. ^--^ 

It seems that after a number of yearjs of R & D in a very complex 
field, you and your 'group have found solutions to numerous problems 
that have stood in the way of the development of reaJJ^^^effective CAI 
systems. I think that as soon as yoi| catch up ofr'some of. your docur 
mentation, the field -of computer-assisted instruction is/foing to 
profit substantially from what you f^ave done and are doing. 

\ . Cordially, • - • - • 

• ^ • \ C. Victor Bunderson - 

/mi ' * ^ Director, „XArLab 
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Appendix F 
FILE ACTIVITY CONtRii 



:STEM (FACS) 



HumRRO Research Product by Leslie Willis, Jean Garrifeau, and 
John Stelzer, Volume V in Proiect IMPACT Software Documantation, 
RP D 1-72-4, August 1972 ■ 



Abstract 

The Project IMPACT File Activity Control System (FACS) is an authoring aid used 
to assist in the development of instructional text FACS provides printouts of textual 
elements in the exact format that they appear to the student on the cathode ray tube. 
FACS also provicies printouts of logical units of instructional elements in compressed 
form: FACS allov^^s an author to perform character string searches on the instructional 
text -files in order to identify elements that contain specified character strings' Instruc- 
tional elements can be stored with administrative data identifying the author of the text 
and the date of preparation. Generally, FACS prints this information with its reports and 
allows modifi?!ation. This report describes the use and operation of the FACS systenC 
Reports produced by FACS are also described, •c , . • _ . 
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Appendix G 

PROJECT l^^P^^;tfo?p/hR^ D.QGUMENTATION:\ixVERVIEV<^P' 
T^CCWUTEtJ-ADAAINJSTERED INSTRUCTIOM SUDSYStEM , 



/ , % HumRRO TechnicarReport 72-21, by Jofyi Stelzer and 
/ ^ Jean Gameau, August 1972 
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Summary 



Military Problem 



The combination of shrinking financial resources and the prpspects of a smaller, 
all-volunteer Army will increase both the demands made on ^ Army personnel an'dJhe 
importance of the individual soldier. There wll be a greater need for more effective and ' 
efficient training, adequate to the task of providing an increasing number of cbipplex 
skills to widely differing students, while using fower skilled instructors' 

.The most promising approach available to m^eet these new training demands is* 
computer-admin istexed instruction (CAI), if it is developed as a comprehensive, 
total system. . . ^ * 

The goal of Project IMPACT is tp provide the Army with an effective, efficient, and \ 
economical -CAI' system in a total system framework. To be effective, the system should 
maximiz^the achievement of the students and the instructors to a greater extent than is 
possible in the traditional classroom; to be efficient, it should providej maximum 
productivity per unit time on the part of instructors, administrators, and students; and to ■ 
be economical, the, cost and^resources must -not exceed those of a comparable effective 
non-CAI ir^^tmctjonal System. t 

Development Problem, and Approach > " 

Ptpject IMP/iCT was established by the Department of the Army in 1968 as an 
advanced development effort to provide a total system of CAI for the effective and ' 
efficient training of niiljtlary personnel. Accordingly^ a^Technical Development Plan (TDP) 
was conceived that provides for the concurrent development of the four facets of a total 
CAI system: instructional content, hardware, software, and instructional decision model 
(IDM). The Project was oi^anized to keep these/Tacets in balance over a span of two 
generations of CAI systems and four successive cycles t>f development and testing. The 
initial two cycles covering? the development and test of a "breadboard'' CAI system have 
been completed. The secpnd two cycles were planned as a period for refinement of all 
facets of the system, to\produce a prototype model to be tested,, evakiated, and then 
delivered to the Army as st)ecificalions for an operational instructional'^y'stem. 

In pursuing its goal,^ Project IMPA'CT has followed an evolutionary approach toward 
developing products usable by Army instructional staff. This document describes the 
overall first generation, IMPACT-A, software products. The intent for widespread Army 
use is to provide functional requirements for a cost/of foctive system. 
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Products 

jFhe documents in .the software series have been prepared to ass^t systems program- 
mers in incorporating all or some of the IMPACT-A software products into on-going CAI 
efforts. While the primary purpose of the initial generation was to develop' and test a 
provisional total CAI system, many of its products, such as the time-sharing software 
data management capabilities, and IDM guidelimes can fulfill us,er needs now. Subsequent 
products from the continued effort (IMPACT-B) will document the revision and refine- 
ments to these items. The software products are: ' \ • 

(1) Zeus Documentation-operationally available\ time-sharing software; author- 
ing command set; separate text and course logic facility. \ 

* ■ (2rFAeS-(file Activity Control System)-a set of computer programs that 

provides information cpnceming display pages that are disk stored; a system to assist in 
editing? and coordinating displays. J. 

(3) IDES-l' (liVIPACT Data Evaluation System-yersion l)-a set of computer 
programs that manage <the dat^i collected, stored, and processed by the CaI system (no 
longer used). - ' 

(4) IDES-2 (IMPACT Data Evaluation System-Version 2)-an updated software 
system that "provides for storage, retrieval, and analysis of student generated data. 

(5) The Interface— a system that maintains on-line records of the prerequisites 
that have been satisfied by each student: it also controls intermodule and intramodule 
transfers. 

^(6) Coursewriter III Functions-a version of IBM's Coursewriter III that 
performs response analyses, da£a generation, and branching for students and data manipu- 
lation capabilities: ■ . ' 

An overview of this software system is presented in -tliis report. The products are 
described in detail in a series of Research Products intc^led primarily for personnel 
working in the computer .software field. - - <» 



ERIC 



- 1 67 



3 



Appendix H ^ 



THE INTERFACE SUBSYSTEM FRAMEWORK FOR 
INSTRUCTIONAL DECISION MOPELING 

HumRRO Research Product by William Underbill and John Stelzer,' 
Volume IV m.Project IMPACT^ Software Documentation, 
RP-D 1.72-3, August 1972 



Abstract 



The IMPACT C^iqrjpu^r-Admi^istered Instruction (CAI) software subsystem utilftjes 
Coursewriter III as its primary vehicle for providing student instruction. IMPACT Course- 
writer III instructional material is structured into divisions, with ^ach division having one 
or more instrUctionaj^iiodules. Each module has a Telling (T) section with a Practice (P) 
subsection, and a Quiz (Q) section. A student may recapitulate- any completed module, 
^vie\y his current module's, T-section,* or jump to. the current module's practice or qtiiz 
sections. System^cheduled remediation is also provided for in* IMPACT'S instruction. The 
Interface controls alt intermodule and intramodule transfers. It is used to assemble the 
appropriate label when a transf^ is mad^. The label is returned to Course writer. I II and is 
used in 'a Coursewriter III branch instructiori. Interface permits an author to specify, for 
each individual student, a separate and unique division and module structure. Thus, it 
ajso allows the author to specify an individual course for ea^h student with the course . 
components being di^wn from a pool of instructional material. 
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Appendix I 

ITARACFS COMPUTER-ADMINISTERED INSTROCTIdN 
.SORWARE SUBSYSTEM, ^C'OURSEWRITER llj, AND ITS FUNCTJONS 

} > HumRRQ Research Product by'Doris Shuford and John Stelzer, 
- ^Volume VII in Project IMPACT Software Documentation) 

; ' RP-D1-72-6, August 1972 ^ \^ 



Abstract ^ - ' 
« » - « 

The. cornputer«^dministered instruction (CAI) lan^mage "component, in Project 
IMCACT's^ CALsystemJs an IBM program product, Coursewiter III Version 2,, which has 
been modified slightly at IMPACT. The modifications concern what data are recorded by 
Coursewriter III and Ho\y and when data are recorded. The modifications also provipled 
for special haild"ling o'f .invalid sign-on attempts, &nd special processing ot commands and 
symbbls not normally recbgnlzed -by Coursewriter III.' IP^tPACT lias also developed and 
h^s in use several Coursewriter III functions for clearing counters and switches, for 
storing and, loading buffers and counter^, fgr special processing of bnffers, for processing 
student response, for recording data, and for performing arithrnetic computations on-line. 
This document provides detailed documentatipn pn'ali'^uAursewriter III modifications 
made at IMPACT and on^ all Coursewriter iirfxinctiofts use4 at IMPACT. 
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Apperjdix J . . 

Hta evaluation system- ' ' . ^ . " 

HumRRO Research Product by Johrlv^elzer and Leslie Willis, . 
Volume niin Project IMPACT Software Documentation,^ 
RP.D172-2; August 1942" ' , . ^ ^ 

Abstract u - ' * 

This report describes the IMPACT Data Evaluation 'System-Version 1 ,(IDES-1); 
EDES-l has two main cbmponents-^torage/retrieval affid ^alysis. The storage /retrieval 
component is used to update and* maintain an extensive. 'data base of Computer- 
Administered Instruction (CAI) gerferated^jiata^^ as well as to retrieve'* selective data 
elements from the data base. The data are used for psychological research in learning, and 
for • evaluating the instructional material. In IDES-1, the storage (retrieval fjinction is 
perfornied through a list processor^, SLIP. IMPACT'S version of SLIP* has been modified 
^ extended for more efficient operation. ThQ analysis function in LMPACT is intended 
to provide statistical analysis of data base subsets. This function is performed through the 
BMD statistical analysis package, augmented, by ^specially prepared programs. This 
document describes in detail the storage and retrieval portion of IDES-l (SLIP itself aud 
the BMD package dre not described). ^ . ' . . 




Appendix K * ... 

THE t/SAPACT DATA EVALUATION SYSTEM- 
^ VERSION 2 (I&ES-2) 

^ HumRRp Research Product by Leslie Willis ahd^hn Stelzer, 
yo\umQ\\'\n Project IMPACT Software Dbcumentath 
RP-Dl-72-1. August 1972 

« 

> • , ' Abstract 

The IMPACT Data Evaluation System— Version 2 (IDES-2) provides a storage, 
retxieval, and analysis capability for data generated in Projects IMPACT'S CAI environ- 
ment. IDES:'2 uses standard. PL/1 techniques to perform the required storage, retrieval, 
and file .maintenance activities. Statistical analysis in IDES-2 is provided through the 

'Biomedical (BMD) statistical analysis package, augmented as required at Project IMPACU 
by especially prepared routine"^. JDES.2 provides exteiieive, standard reports summarizing 
student activity, which are used by authors to. evaluate the effectiveness of the 

- instructional material. IDES-2 reports are also used by iMPACT^s research personnel to 
monitor student activity.^ As a result, IMPACT is able to develop increasingly more 
efficient instructional ;lecision' models. The storage and retrieval component m iPES-2 is 
documented in detail in "this document. IDES.2 reports, including the method thrpugh 
which each Irepor^t is generated and its contents are described, and examples of reports, 
are provided. ♦ , " ^ 




